{"data":{"slug":"philolabs-agentic-vbench","name":"agentic-vbench","tagline":"A benchmark for evaluating AI agents in performing real-world post-production tasks like audio and video editing.","github_url":"https://github.com/PhiloLabs/agentic-vbench","owner":"PhiloLabs","repo":"agentic-vbench","owner_avatar_url":"https://avatars.githubusercontent.com/u/260782224?v=4","primary_language":"Python","stars":96,"forks":27,"topics":["ai-agents","benchmark","harbor","llm-evaluation","video-editing"],"archived":false,"github_pushed_at":"2026-09-02T18:05:36+00:00","maintenance_label":"Very active","stars_delta_30d":14,"url":"https://www.graphcanon.com/tools/philolabs-agentic-vbench","markdown_url":"https://www.graphcanon.com/tools/philolabs-agentic-vbench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/philolabs-agentic-vbench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=philolabs-agentic-vbench","description":"AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?","homepage_url":"https://agenticvbench.com/","license":"Apache-2.0","open_issues":37,"watchers":0,"ai_summary":"AgenticVBench evaluates AI agent capabilities to handle real-world post-production tasks such as audio restoration by measuring performance metrics, processing time, and cost. It provides specific task prompts that guide the agents through various restorative processes.","readme_excerpt":"### 1. Install\n\n```bash\ngit clone https://github.com/PhiloLabs/agentic-vbench.git\ncd agentic-vbench\n./scripts/install-harbor.sh\npython3 -m venv .venv && .venv/bin/pip install --upgrade pip\n```\n\n---\n\n# reward.json → ≈ 1.0, ~30 s on a cached image, zero agent cost\n```\n\n**Time + cost budget.** A real-agent rollout on Modal typically takes ~10 min per task wall clock. Cost depends entirely on agent + model — order-of-magnitude $0.10–$2 per task with mid-tier models, scaling linearly with the agent's token use. Plan accordingly for a 100-task sweep.\n\n**Here's what a task prompt actually looks like** (`exp-codec-restore-task01`):\n\n> # Restore A Muffled Stretch Of Audio\n>\n> I have a short mono speech recording at `/workspace/materials/noisy.wav`. For a stretch in it, the audio sounds muffled — like the high end has been chopped off and the voice lost its sparkle. The rest of the recording sounds clean and full.\n>\n> Please restore the muffled stretch so it sounds as clear and full as the rest of the recording. Leave the already-clean parts unchanged.\n>\n> ## What to deliver\n> - `/workspace/output/enhanced.wav` — 16-bit PCM mono at 16 kHz, same total length (sample count) as the input.\n\nEach task ships its own such brief at `tasks/<family>/<task>/steps/solve/instruction.md`.\n\n**Inspect the result.** Each trial drops four artifacts under `jobs/<job-name>/<trial-id>/`:\n\n| File | What it is |\n|---|---|\n| `steps/solve/verifier/reward.json` | Final score + per-metric breakdown. |\n| `agent/trajectory.json` | Full event stream Harbor captured for the agent (tool calls, tool results, model messages, final output). |\n| `result.json` | Per-trial Harbor summary (timings, exit codes, exception info). |\n| `trial.log` | Combined stdout/stderr stream for the whole trial. |\n\n```bash\n./avb results show          # rewards from the latest job\ncat jobs/<job-name>/*/steps/solve/verifier/reward.json","github_created_at":"2026-05-11T20:13:06+00:00","created_at":"2026-07-15T10:40:15.845621+00:00","updated_at":"2026-09-20T04:25:07.985333+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"ai-agents","name":"ai-agents"},{"slug":"benchmark","name":"benchmark"},{"slug":"harbor","name":"harbor"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"video-editing","name":"video-editing"}],"trust":{"provenance":{"is_fork":false,"github_id":1235951137,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-09T06:00:50.588Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":6,"last_release_at":"2026-05-23T08:17:37Z","stars_delta_30d":14,"open_issues_delta_30d":-20},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:40:17.191Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-09T06:00:51.201Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-09T06:00:51.201Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-09T06:00:51.201Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Install via scripts provided in the repository.","Python virtual environment setup for reproducibility."],"requires_docker":true},"constraints":{"requires_docker":true},"when_to_use":["When you need to benchmark the performance of AI agents in handling specialized tasks such as audio and video editing that require precise restorative actions.","If your development team requires metrics on processing time, cost, and accuracy when evaluating different AI agent algorithms designed for post-production applications."],"when_not_to_use":["When the focus is on generic performance evaluations rather than on real-world, task-specific benchmarks that assess handling complex post-production scenarios.","If your budget or timeline cannot accommodate a per-task wall clock time of ~10 minutes and cost ranging from $0.10 to $2 based on agent token usage."],"source":"enrich:decision_facts","observed_at":"2026-07-16T19:14:25.075Z"},"constraint_facets":{"requires_docker":true},"decision_summary":[{"label":"Requirements","value":"Requires Docker; Install via scripts provided in the repository.; Python virtual environment setup for reproducibility."},{"label":"Adopt for","value":"AgenticVBench evaluates AI agents' real-world post-production capabilities with specific task prompts for activities like audio restoration."}]}}