{"data":{"slug":"avarok-cybersecurity-atlas","name":"atlas","tagline":"Pure Rust Inference Engine","github_url":"https://github.com/Avarok-Cybersecurity/atlas","owner":"Avarok-Cybersecurity","repo":"atlas","owner_avatar_url":"https://avatars.githubusercontent.com/u/96390941?v=4","primary_language":"Rust","stars":697,"forks":105,"topics":["cuda","dgx","dgx-spark","gb10","llm-inference","mamba","nvfp4","openai-api","rust","speculative-decoding","ssm","transformers"],"archived":false,"github_pushed_at":"2026-09-20T03:33:55+00:00","maintenance_label":"Very active","stars_delta_30d":87,"url":"https://www.graphcanon.com/tools/avarok-cybersecurity-atlas","markdown_url":"https://www.graphcanon.com/tools/avarok-cybersecurity-atlas.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/avarok-cybersecurity-atlas","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=avarok-cybersecurity-atlas","description":"Pure Rust Inference Engine","homepage_url":"https://atlasinference.io","license":"AGPL-3.0","open_issues":237,"watchers":7,"ai_summary":"Avarok-Cybersecurity/atlas is a inference engine written in Rust, focused on providing efficient and scalable model deployment.","readme_excerpt":"### Docker\n\nThe whole supported model matrix lives in one image. Pull it, mount your HuggingFace cache, and point `serve` at any model ID from the [model table](#models).\n\n> [!TIP]\n> The recipes below are tuned for **maximum accuracy under agentic-coding workloads** — 64K context, BF16 MTP draft head (highest acceptance rate ⇒ highest end-to-end throughput), prefix caching for multi-turn tool loops, and FP8 KV cache with `auto`-promoted boundary layers. These are the exact configurations we use to drive opencode / Claude Code / Cline through Atlas on a single Spark.\n\n#### Recipe 0 — no flags, pick a model in the TUI\n\nOmit the model ID and `serve` boots into the Library — pick a model and recipe interactively (TTY only):\n\n```bash\ndocker run -it --rm --network host --gpus all --ipc=host \\\n  -v \"${HOME}/.cache/huggingface:/root/.cache/huggingface\" \\\n  -v \"${HOME}/.avarok:/root/.avarok\" \\\n  avarok/atlas-gb10:latest serve\n```\n\n- `-it` — the TUI needs a real terminal to render (and Esc to quit).\n- `--rm` — throwaway container; nothing to clean up after the session.\n- `--network host` — the served port is reachable on localhost directly, no `-p` mapping.\n- `--gpus all` — hands the GB10 to the container.\n- `--ipc=host` — host-sized shared memory; the Docker default 64 MB `/dev/shm` is too small for CUDA.\n- `-v ~/.cache/huggingface` — reuse the host's model cache instead of re-downloading weights.\n- `-v ~/.avarok` — persist Atlas state (recipes, benchmark records, artifacts) across runs.\n\n<a id=\"run-atlas\"></a>\n\n#### Recipe A — Qwen3.6-35B-A3B (FP8 hybrid MoE, the daily driver)\n\n35 B params, 3 B active, GDN + attention + 256-expert MoE, MRoPE-positioned vision tower (text-only here).\n\n```bash\ndocker pull avarok/atlas-gb10:latest\n\nsudo docker run -d --name atlas \\\n  --network host --gpus all --ipc=host \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  avarok/atlas-gb10:latest \\\n  serve Qwen/Qwen3.6-35B-A3B-FP8 \\\n    --port 8888 \\\n    --max-seq-len 65536 \\\n    --kv-cache-dtype fp8 \\\n    --kv-high-precision-layers auto \\\n    --gpu-memory-utilization 0.90 \\\n    --scheduling-policy slai \\\n    --enable-prefix-caching \\\n    --speculative \\\n    --num-drafts 2 \\\n    --tool-call-parser qwen3_coder\n```\n\nWhy these flags:\n\n- `--max-seq-len 65536` — 64K window for long agent traces, file reads, multi-step tool use.\n- `--kv-cache-dtype fp8 --kv-high-precision-layers auto` — half the memory of BF16, no measurable quality loss; the boundary attention blocks stay BF16, where the routing distribution is most sensitive. `auto` is **not** a heuristic — it is a fixed alias for `2` (`serve_phases/kv_cache.rs`), alongside `max`/`all` meaning \"every attention layer\". Because it is non-zero it also *suppresses* the per-dtype automatic promotion that `0` would trigger under a `turbo*` KV dtype.\n- `--scheduling-policy slai` — SLAi scheduler. **Not the default** — `serve` defaults to `fifo`, so this flag has to be passed to get SLO-aware ordering. It reorders concurrent sequences to keep MTP verify batches dense and prefills shortest-prompt-first.\n- `--enable-prefix-caching` — radix-tree prefix cache; tool-use sessions reuse the system prompt + tool-defs + earlier turns.\n- `--speculative --num-drafts 2` — MTP draft head proposes 2 tokens per step. **No `--mtp-quantization` flag** ⇒ defaults to **BF16**, which gives the highest acceptance rate (lossier MTP projections lower acceptance and usually *worsen* end-to-end tok/s, despite the faster draft forward).\n- `--tool-call-parser qwen3_coder` — explicit Qwen XML tool format. Atlas auto-resolves the right parser from `tool_defaults.toml` per model; pass it anyway in production scripts.\n\n#### Recipe B — Qwen3.5-35B-A3B (NVFP4, ~131 tok/s with MTP K=2)\n\nThe fastest model in the matrix on a single Spark.\n\n```bash\nsudo docker run -d --name atlas \\\n  --network host --gpus all --ipc=host \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  avarok/atlas-gb10:latest \\\n  serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \\\n    --po","github_created_at":"2026-05-05T16:59:33+00:00","created_at":"2026-07-11T11:44:57.421508+00:00","updated_at":"2026-09-20T06:35:02.058202+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cuda","name":"cuda"},{"slug":"dgx","name":"dgx"},{"slug":"dgx-spark","name":"dgx-spark"},{"slug":"gb10","name":"gb10"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"mamba","name":"mamba"},{"slug":"nvfp4","name":"nvfp4"},{"slug":"openai-api","name":"openai-api"}],"trust":{"provenance":{"is_fork":false,"github_id":1230084743,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T03:43:57.160Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":30,"days_since_push":0,"last_release_at":"2026-09-20T03:33:55Z","stars_delta_30d":87,"open_issues_delta_30d":168},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:44:59.011Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T03:43:58.174Z"},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-09-20T03:43:58.174Z"},"license_spdx":{"value":"AGPL-3.0","source":"github.license","observed_at":"2026-09-20T03:43:58.174Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When aiming for high-performance Rust-based deployment that leverages hardware accelerators like NVIDIA DGX systems and Cuda technology.","If your project requires integration with the OpenAI API or supports Speculative Decoding techniques."],"when_not_to_use":["Avoid if you prefer tools in languages other than Rust for inference engines, since this is purely designed in Rust.","Not ideal if your deployment environment does not support NVIDIA GPU technologies such as DGX, which are key to maximize performance with Atlas."],"source":"enrich:decision_facts","observed_at":"2026-07-14T19:42:11.717Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Focuses on efficient and scalable model deployment with Rust, supporting various GPU technologies and inference frameworks."}]}}