{"data":{"slug":"alphadl-adarubrics","name":"AdaRubrics","tagline":"Adaptive Dynamic Rubric Evaluator for Agent Trajectories","github_url":"https://github.com/alphadl/AdaRubrics","owner":"alphadl","repo":"AdaRubrics","owner_avatar_url":"https://avatars.githubusercontent.com/u/20458732?v=4","primary_language":"Python","stars":362,"forks":37,"topics":["agent-evaluation","llm-evaluation","reward-model","rlhf","rubric"],"archived":false,"github_pushed_at":"2026-08-24T06:33:01+00:00","maintenance_label":"Active","stars_delta_30d":17,"url":"https://www.graphcanon.com/tools/alphadl-adarubrics","markdown_url":"https://www.graphcanon.com/tools/alphadl-adarubrics.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/alphadl-adarubrics","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=alphadl-adarubrics","description":"AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":24,"ai_summary":"AdaRubric is an evaluation tool designed to assess the performance of AI agents and language models based on dynamic rubrics that adapt according to agent trajectories.","readme_excerpt":"## Installation\n\n```bash\ngit clone https://github.com/alphadl/AdaRubrics.git\ncd AdaRubrics\npip install -e \".[dev]\"\n```\n\nSet `OPENAI_API_KEY` in your environment (or pass via config). YAML config support requires `pip install pyyaml`.\n\n---\n\n---\n\n## Quick Start\n\n```python\nimport asyncio\nfrom adarubric import AdaRubricPipeline, TaskDescription, Trajectory, TrajectoryStep\nfrom adarubric.config import AdaRubricConfig\n\ntask = TaskDescription(\n    task_id=\"demo-001\",\n    instruction=(\n        \"Use the weather API to check if it will rain in Tokyo tomorrow, \"\n        \"and if so, suggest indoor activities.\"\n    ),\n    domain=\"Personal Assistant\",\n    expected_tools=[\"weather_api\", \"activity_search\"],\n)\n\ntrajectory = Trajectory(\n    trajectory_id=\"traj-demo-001\",\n    task_id=\"demo-001\",\n    steps=[\n        TrajectoryStep(\n            step_id=0,\n            thought=\"I need to check tomorrow's weather in Tokyo first.\",\n            action=\"weather_api\",\n            action_input={\"city\": \"Tokyo\", \"date\": \"tomorrow\"},\n            observation=\"Tomorrow: 70% chance of rain, high 18°C, low 12°C.\",\n        ),\n        TrajectoryStep(\n            step_id=1,\n            thought=\"It's likely to rain. Let me find indoor activities.\",\n            action=\"activity_search\",\n            action_input={\"city\": \"Tokyo\", \"type\": \"indoor\", \"limit\": 5},\n            observation=\"1. TeamLab Borderless, 2. Tokyo National Museum, 3. Akihabara arcades...\",\n        ),\n    ],\n)\n\npipeline = AdaRubricPipeline.from_config(AdaRubricConfig())\nresult = asyncio.run(pipeline.run(task, [trajectory], num_dimensions=5))\n\nprint(f\"Rubric dimensions: {result.rubric.dimension_names}\")\nprint(f\"Global score:      {result.mean_score:.2f}/5.0\")\nprint(f\"Survival rate:     {result.survival_rate:.0%}\")\n```\n\nRun the full example:\n\n```bash\nexport OPENAI_API_KEY=\"sk-...\"\npython examples/quickstart.py\n```\n\n---","github_created_at":"2026-02-22T19:45:16+00:00","created_at":"2026-07-11T12:00:21.098262+00:00","updated_at":"2026-09-20T06:58:58.846799+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agent-evaluation","name":"agent-evaluation"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"reward-model","name":"reward-model"},{"slug":"rlhf","name":"rlhf"},{"slug":"rubric","name":"rubric"}],"trust":{"provenance":{"is_fork":false,"github_id":1164205568,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T06:58:56.023Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":27,"last_release_at":null,"stars_delta_30d":17,"open_issues_delta_30d":1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:00:22.380Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T06:58:57.278Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T06:58:57.278Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T06:58:57.278Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need dynamic evaluation criteria that adapt in real-time according to how your AI agents or language models are performing their tasks.","If your project involves multiple stages of AI agent development where continuous adaptation of evaluation metrics is necessary for improving performance over time."],"when_not_to_use":["If fixed rubrics with static evaluation criteria suffice, AdaRubrics provides more complexity than needed.","For projects that do not require real-time adjustments in evaluation methods as the AI agents' or models' trajectories progress."],"source":"enrich:decision_facts","observed_at":"2026-07-17T11:09:19.920Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"AdaRubrics serves as an Adaptive Dynamic Rubric Evaluator specifically for assessing AI agent and language model performance based on evolving rubrics tailored to the agents' paths."}]}}