Artificial Intelligence Controller Interface (AICI)

LLGuidance library is an actively maintained evolution and specialization of AICI, recommended if all you want is constrained decoding.

The Artificial Intelligence Controller Interface (AICI) lets you build Controllers that constrain and direct output of a Large Language Model (LLM) in real time. Controllers are flexible programs capable of implementing constrained decoding, dynamic editing of prompts and generated text, and coordinating execution across multiple, parallel generations. Controllers incorporate custom logic during the token-by-token decoding and maintain state during an LLM request. This allows diverse Controller strategies, from programmatic or query-based decoding to multi-agent conversations to execute efficiently in tight integration with the LLM itself.

The purpose of AICI is to make it easy to build and experiment with both existing and entirely new Controller strategies for improving LLM generations. By abstracting away implementation details of the underlying LLM inference and serving engine, AICI aims to simplify the development of Controllers, make it easier to write fast Controllers, and ease compatibility across LLM inference and serving engines.

AICI is designed for both local and cloud execution, including (eventually) multi-tenant LLM deployments. Controllers are implemented as light-weight WebAssembly (Wasm) modules which run on the same machine as the LLM inference engine, utilizing the CPU while the GPU is busy with token generation. AICI is one layer in the inference stack, and is designed to allow control libraries such as Guidance, LMQL, and others to run on top of it and gain both efficiency and performance improvements, as well as portability across LLM inference and serving engines.

AICI currently integrates with llama.cpp, HuggingFace Transformers, and rLLM (custom tch-based LLM inference engine), with vLLM in the works.

AICI is:

• Flexible: Controllers can be written in any language that can compile to Wasm (Rust, C, C++, ...), or be interpreted inside Wasm (Python, JavaScript, ...)
• Secure: Controllers are sandboxed and cannot access the filesystem, network, or any other resources
• Fast: Wasm modules are compiled to native code and run in parallel with the LLM inference engine, inducing only a minimal overhead to the generation process

AICI is a prototype, designed and built at Microsoft Research.

Table of Contents

• Artificial Intelligence Controller Interface (AICI)
• QuickStart: Example Walkthrough
  • Development Environment Setup
  • Build and start rLLM server and AICI Runtime
  • Control AI output using AICI controllers
• Comprehensive Guide: Exploring Further
• Architecture
• Security
• Performance
• Flexibility
• Acknowledgements
• Contributing
• Trademarks

QuickStart: Example Walkthrough

In this quickstart, we'll guide you through the following steps:

• Set up rLLM Server and AICI Runtime.
• Build and deploy a Controller.
• Use AICI to control LLM output, so you can customize a LLM to follow specific rules when generating text.

Development Environment Setup

To compile AICI components, you need to set up your development environment for Rust. For this quickstart you also need Python 3.11 or later to create a controller.

Windows WSL / Linux / macOS

[!NOTE] Windows users: please use WSL2 or the included devcontainer. Adding native Wind