{"data":{"slug":"notpunchnox-rkllama","name":"rkllama","tagline":"Ollama alternative for Rockchip NPU with optimized AI and Deep learning model inference","github_url":"https://github.com/NotPunchnox/rkllama","owner":"NotPunchnox","repo":"rkllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/74469788?v=4","primary_language":"Python","stars":590,"forks":99,"topics":["ai","client","client-server","ia","llm","llm-apps","llm-inference","npu","npu-llm","offline","orange-pi","orangepi","orangepi5","orangepi5pro","python","rk3576","rk3588","rockchip","server"],"archived":false,"github_pushed_at":"2026-07-07T06:49:18+00:00","maintenance_label":"Steady","stars_delta_30d":13,"url":"https://www.graphcanon.com/tools/notpunchnox-rkllama","markdown_url":"https://www.graphcanon.com/tools/notpunchnox-rkllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/notpunchnox-rkllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=notpunchnox-rkllama","description":"Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )","homepage_url":null,"license":"GPL-3.0","open_issues":65,"watchers":21,"ai_summary":"This repository provides an efficient solution for running AI and deep learning models on Rockchip devices, featuring enhanced support for Rockchip Neural Processing Unit (NPU) through rkllm.","readme_excerpt":"## Tested Hardware and Environment\n- **Hardware**: Orange Pi 5 Pro: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.\n- **Hardware**: Orange Pi 5 Plus: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.\n- **Hardware**: Orange Pi 5 Max: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.\n- **Hardware**: Radxa Rock 4d: (Rockchip RK3576, NPU 6 TOPS), 16GB RAM.\n- **OS**: [Ubuntu 24.04 arm64.](https://joshua-riek.github.io/ubuntu-rockchip-download/)\n- **OS**: Armbian Linux 6.1.99-vendor-rk35xx (Debian stable bookworm), v25.2.2.\n\n---\n\n###  Standard Installation (recommended create a virtual environment like: conda, uv, venv)\n\n1. **Clone the repository:**\n\n```bash\ngit clone https://github.com/notpunchnox/rkllama\ncd rkllama\n```\n\n2.  **Install RKLLama:**\n\n```bash\npython -m pip install .\n```\n\n**Output:**\n\n---\n\n### Docker Installation\n\nPull the RKLLama Docker image:\n\n```bash\ndocker pull ghcr.io/notpunchnox/rkllama:main\n```\nrun server\n```bash\ndocker run -it --privileged -p 8080:8080 -v <local_models_dir>:/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main \n```\n\n*Set up by: [ichlaffterlalu](https://github.com/ichlaffterlalu)*\n\n#### Docker Compose\n\nDocker Compose facilities much of the extra flags declaration such as volumes:\n\n```bash\ndocker compose up --detach --remove-orphans\n```\n\n---\n\n## Tool Calling Quick Start\n\nRKLLama supports advanced tool/function calling for enhanced AI interactions:\n\n```bash\n\n---\n\n### **Manual Installation**\n1. **Download the Model**\n   - Download `.rkllm` models directly from [Hugging Face](https://huggingface.co).\n   - Alternatively, convert your GGUF models into `.rkllm` format (conversion tool coming soon on [my GitHub](https://github.com/notpunchnox)).\n\n2. **Place the Model**\n   - Create the `models` directory on your system.\n   - Make a new subdirectory with model name.\n   - Place the `.rkllm` files in this directory.\n   - Create `Modelfile` and add this :\n\n   ```env\n    FROM=\"file.rkllm\"\n    HUGGINGFACE_PATH=\"huggingface_repository\"\n    SYSTEM=\"Your system prompt\"\n    TEMPERATURE=1.0\n    ```\n\n   Example directory structure:\n   ```\n   ~/RKLLAMA/models/\n       └── TinyLlama-1.1B-Chat-v1.0\n           |── Modelfile\n           └── TinyLlama-1.1B-Chat-v1.0.rkllm\n   ```\n\n   *You must provide a link to a HuggingFace repository to retrieve the tokenizer and chattemplate. An internet connection is required for the tokenizer initialization (only once), and you can use a repository different from that of the model as long as the tokenizer is compatible and the chattemplate meets your needs. Tokenizer gets downloaded for the first time in the models directory*\n\n---\n\n### **For Multimodal Encoder Model (.rknn) Installation**\n1. **Download the encoder model .rknn**\n   - Download `.rknn` models directly from [Hugging Face](https://huggingface.co).\n   - Alternatively, convert your ONNX models into `.rknn` format.\n   - Place the `.rknn` model inside the `models` directory. RKLLama detected the encoder model present in the directory.\n   - Include manually the following properties in the `Modelfile` according to the conversion properties used for the conversion of the vision encoder `.rknn`:\n   ```env\n    IMAGE_WIDTH=448\n    IMAGE_HEIGHT=\n    N_IMAGE_TOKENS=\n    IMG_START=\n    IMG_END=\n    IMG_CONTENT=\n\n    # For example, for Qwen2VL/Qwen2.5VL:\n\n    IMAGE_WIDTH=392\n    IMAGE_HEIGHT=392\n    N_IMAGE_TOKENS=196\n    IMG_START=<|vision_start|>\n    IMG_END=<|vision_end|>\n    IMG_CONTENT=<|image_pad|>\n\n    # For example, for MiniCPMV4:\n\n    IMAGE_WIDTH=448\n    IMAGE_HEIGHT=448\n    N_IMAGE_TOKENS=64\n    IMG_START=<image>\n    IMG_END=</image>\n    IMG_CONTENT=<unk>\n   ```\n\nExample directory structure for multimodal:\n   ```\n   ~/RKLLAMA/models/\n       └── qwen2-vision\\:2b\n           |── Modelfile\n           └── Qwen2-VL-2B-Instruct.rkllm\n           └── Qwen2-VL-2B-Instruct.rknn\n   ```\n\n---\n\n### **For Image Generation Installation**\n1. In a temporary folder, clone the repository https://huggingface.co/danielferr85/lcm-sd-1.5-rknn-2.3.2-rk3588 or https://huggingface","github_created_at":"2024-12-30T22:47:39+00:00","created_at":"2026-07-11T11:45:13.09803+00:00","updated_at":"2026-08-25T06:02:03.180953+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"client-server","name":"client-server"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"npu-llm","name":"npu-llm"},{"slug":"orange-pi","name":"orange-pi"},{"slug":"orangepi5pro","name":"orangepi5pro"},{"slug":"python","name":"python"},{"slug":"rk3576","name":"rk3576"}],"trust":{"provenance":{"is_fork":false,"github_id":910293344,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T06:02:02.431Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":48,"last_release_at":"2025-03-24T18:46:33Z","stars_delta_30d":13,"open_issues_delta_30d":6},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:45:14.459Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T06:02:02.857Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-25T06:02:02.857Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-25T06:02:02.857Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-25T06:02:02.857Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-25T06:02:02.857Z"},"license_spdx":{"value":"GPL-3.0","source":"github.license","observed_at":"2026-08-25T06:02:02.857Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You need to run models specifically optimized for Rockchip Neural Processing Unit (NPU)","Your deployment involves Rockchip hardware like rk3576, rk3588, Orange Pi 5 Pro"],"when_not_to_use":["Your hardware does not include a Rockchip NPU","You are looking for an AI solution that works across multiple non-Rockchip platforms"],"source":"enrich:decision_facts","observed_at":"2026-07-16T20:20:27.967Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Ollama alternative for Rockchip NPU: optimized AI and deep learning inference on Rockchip devices"}]}}