Cactus

[![Docs][docs-shield]][docs-url] [![Website][website-shield]][website-url] [![GitHub][github-shield]][github-url] [![HuggingFace][hf-shield]][hf-url] [![Reddit][reddit-shield]][reddit-url] [![Blog][blog-shield]][blog-url]

A hybrid edge-cloud AI engine for mobile devices & wearables.

┌─────────────────┐
│  Cactus Engine  │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘     
         │
┌─────────────────┐
│  Cactus Graph   │ ←── Zero-copy computation graph
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Kernels  │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.)
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Quants   │ ←── Custom rotation-based quantization technique
└─────────────────┘  

Quick Demo (Mac)

• Step 1: brew install cactus-compute/cactus/cactus
• Step 2: cactus run

Cactus Engine

#include "cactus_engine.h"

cactus_model_t model = cactus_init(
    "path/to/weight/folder",
    "path to txt or dir of txts for auto-rag",
    false
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";

const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";

char response[4096];
int result = cactus_complete(
    model,            // model handle
    messages,         // JSON chat messages
    response,         // response buffer
    sizeof(response), // buffer size
    options,          // generation options
    nullptr,          // tools JSON
    nullptr,          // streaming callback
    nullptr,          // user data
    nullptr,          // pcm audio buffer
    0                 // pcm buffer size
);

Example response from Gemma4-E2B

{
    "success": true,        // generation succeeded
    "error": null,          // error details if failed
    "cloud_handoff": false, // true if cloud model used
    "response": "Hi there!",
    "function_calls": [],   // parsed tool calls
    "segments": [],         // transcription segments (empty for chat)
    "confidence": 0.8193,   // model confidence
    "confidence_threshold": 0.7, // resolved handoff threshold (model-dependent)
    "time_to_first_token_ms": 45.23,
    "total_time_ms": 163.67,
    "prefill_tps": 1621.89,
    "decode_tps": 168.42,
    "ram_usage_mb": 245.67,
    "prefill_tokens": 28,
    "decode_tokens": 50,
    "total_tokens": 78
}

Cactus Graph

#include "cactus_graph.h"

CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);

auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);

float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};

graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);

graph.execute();
void* output_data = graph.get_output(result);

graph.hard_reset(); 

Inference Speed

• LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
• VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
• Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
• 1k-Context RAM: peak MB during the LLM benchmark
• No speculative decode or MTP, pure decode

Command: cactus benchmark [optional --ios or --android]