{"data":{"slug":"microsoft-aici","name":"aici","tagline":"Builds Controllers for Constrained LLM Output in Real-time Using Wasm","github_url":"https://github.com/microsoft/aici","owner":"microsoft","repo":"aici","owner_avatar_url":"https://avatars.githubusercontent.com/u/6154722?v=4","primary_language":"Rust","stars":2075,"forks":85,"topics":["ai","inference","language-model","llm","llm-framework","llm-inference","llm-serving","llmops","model-serving","rust","transformer","wasm","wasmtime"],"archived":false,"github_pushed_at":"2025-01-22T21:14:57+00:00","maintenance_label":"Dormant","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/microsoft-aici","markdown_url":"https://www.graphcanon.com/tools/microsoft-aici.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/microsoft-aici","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=microsoft-aici","description":"AICI: Prompts as (Wasm) Programs","homepage_url":null,"license":"MIT","open_issues":41,"watchers":21,"ai_summary":"AICI enables the creation of real-time controllers that constrain and direct Large Language Model (LLM) output via WebAssembly modules running alongside the LLM inference engine.","readme_excerpt":"# Artificial Intelligence Controller Interface (AICI)\n\n**[LLGuidance library](https://github.com/guidance-ai/llguidance) is an actively maintained evolution and specialization of AICI, recommended if all you want is constrained decoding.**\n\nThe Artificial Intelligence Controller Interface (AICI) lets you build Controllers that constrain and direct output of a Large Language Model (LLM) in real time.\nControllers are flexible programs capable of implementing constrained decoding, dynamic editing of prompts and generated text, and coordinating execution across multiple, parallel generations.\nControllers incorporate custom logic during the token-by-token decoding and maintain state during an LLM request. This allows diverse Controller strategies, from programmatic or query-based decoding to multi-agent conversations to execute efficiently in tight integration with the LLM itself.\n\n**The purpose of AICI is to make it easy to build and experiment with both existing and entirely new Controller strategies for improving LLM generations.**\nBy abstracting away implementation details of the underlying LLM inference and serving engine, AICI aims to simplify the development of Controllers, make it easier to \nwrite fast Controllers, and ease compatibility across LLM inference and serving engines.\n\nAICI is designed for both local and cloud execution, including (eventually) multi-tenant LLM deployments.\nControllers are implemented as light-weight WebAssembly (Wasm) modules which run on the same machine as the LLM inference engine, utilizing the CPU while the GPU is busy with token generation.\nAICI is one layer in the inference stack, and is designed to allow control libraries such as Guidance, LMQL, and others to run on top of it and gain both efficiency and performance improvements, as well as portability across LLM inference and serving engines.\n\nAICI currently integrates with llama.cpp, HuggingFace Transformers, and rLLM (custom tch-based LLM inference engine), with vLLM in the works.\n\nAICI is:\n\n- [Flexible](#flexibility): Controllers can be written in any language that can compile to Wasm (Rust, C, C++, ...),\n  or be interpreted inside Wasm (Python, JavaScript, ...)\n- [Secure](#security): Controllers are sandboxed and cannot access the filesystem, network, or any other resources\n- [Fast](#performance): Wasm modules are compiled to native code and run in parallel with the LLM inference engine, inducing only a\n  minimal overhead to the generation process\n\nAICI is a prototype, designed and built at [Microsoft Research](https://www.microsoft.com/en-us/research/).\n\n# Table of Contents\n\n- [Artificial Intelligence Controller Interface (AICI)](#artificial-intelligence-controller-interface-aici)\n- [QuickStart: Example Walkthrough](#quickstart-example-walkthrough)\n  - [Development Environment Setup](#development-environment-setup)\n  - [Build and start rLLM server and AICI Runtime](#build-and-start-rllm-server-and-aici-runtime)\n  - [Control AI output using AICI controllers](#control-ai-output-using-aici-controllers)\n- [Comprehensive Guide: Exploring Further](#comprehensive-guide-exploring-further)\n- [Architecture](#architecture)\n- [Security](#security)\n- [Performance](#performance)\n- [Flexibility](#flexibility)\n- [Acknowledgements](#acknowledgements)\n- [Contributing](#contributing)\n- [Trademarks](#trademarks)\n\n# QuickStart: Example Walkthrough\n\nIn this quickstart, we'll guide you through the following steps:\n\n* Set up **rLLM Server** and **AICI Runtime**.\n* Build and deploy a **Controller**.\n* Use AICI to control LLM output, so you can **customize a LLM to follow specific rules** when generating text.\n\n## Development Environment Setup\n\nTo compile AICI components, you need to set up your development environment for Rust. For this quickstart you also need Python 3.11 or later to create a controller.\n\n### Windows WSL / Linux / macOS\n\n> [!NOTE]\n> **Windows users**: please use WSL2 or the included [devcontainer](https://containers.dev). Adding native Wind","github_created_at":"2023-09-26T21:33:16+00:00","created_at":"2026-07-07T17:42:46.212947+00:00","updated_at":"2026-08-20T18:01:07.567522+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"inference","name":"inference"},{"slug":"language-model","name":"language-model"},{"slug":"llm","name":"llm"},{"slug":"llm-framework","name":"llm-framework"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"rust","name":"rust"}],"trust":{"provenance":{"is_fork":false,"github_id":697007681,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-20T18:01:06.673Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":574,"last_release_at":"2024-04-29T20:31:19Z","stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:21:51.910Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-20T18:01:07.203Z"},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-08-20T18:01:07.203Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-20T18:01:07.203Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium"},"requirements":{"notes":["Requires a compatible LLM inference engine such as llama.cpp, HuggingFace Transformers, or rLLM."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"when_to_use":["When you need to implement secure, fast, and flexible real-time control over the output of large language models via WebAssembly modules.","For projects where you want tight integration between custom control logic and the underlying LLM inference engine to process text generation dynamically."],"when_not_to_use":["If your use case requires only constrained decoding without additional control logic, consider using LLGuidance library as a more specialized alternative.","Avoid if you are looking for a less experimental solution, since AICI is noted as a prototype and may not fit production scenarios requiring stability."],"source":"enrich:decision_facts","observed_at":"2026-07-14T21:23:12.318Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium"},{"label":"Requirements","value":"Requires a compatible LLM inference engine such as llama.cpp, HuggingFace Transformers, or rLLM."},{"label":"Adopt for","value":"Builds real-time controllers using Wasm modules for directing LLM output securely and efficiently."},{"label":"License detail","value":"MIT"}]}}