rkllama

Tested Hardware and Environment

• Hardware: Orange Pi 5 Pro: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Orange Pi 5 Plus: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Orange Pi 5 Max: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Radxa Rock 4d: (Rockchip RK3576, NPU 6 TOPS), 16GB RAM.
• OS: Ubuntu 24.04 arm64.
• OS: Armbian Linux 6.1.99-vendor-rk35xx (Debian stable bookworm), v25.2.2.

Standard Installation (recommended create a virtual environment like: conda, uv, venv)

1. Clone the repository:

git clone https://github.com/notpunchnox/rkllama
cd rkllama

2. Install RKLLama:

python -m pip install .

Output:

Docker Installation

Pull the RKLLama Docker image:

docker pull ghcr.io/notpunchnox/rkllama:main

run server

docker run -it --privileged -p 8080:8080 -v <local_models_dir>:/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main 

Set up by: ichlaffterlalu

Docker Compose

Docker Compose facilities much of the extra flags declaration such as volumes:

docker compose up --detach --remove-orphans

Tool Calling Quick Start

RKLLama supports advanced tool/function calling for enhanced AI interactions:

---

### **Manual Installation**
1. **Download the Model**
   - Download `.rkllm` models directly from [Hugging Face](https://huggingface.co).
   - Alternatively, convert your GGUF models into `.rkllm` format (conversion tool coming soon on [my GitHub](https://github.com/notpunchnox)).

2. **Place the Model**
   - Create the `models` directory on your system.
   - Make a new subdirectory with model name.
   - Place the `.rkllm` files in this directory.
   - Create `Modelfile` and add this :

   ```env
    FROM="file.rkllm"
    HUGGINGFACE_PATH="huggingface_repository"
    SYSTEM="Your system prompt"
    TEMPERATURE=1.0
    ```

   Example directory structure:

~/RKLLAMA/models/ └── TinyLlama-1.1B-Chat-v1.0 |── Modelfile └── TinyLlama-1.1B-Chat-v1.0.rkllm

*You must provide a link to a HuggingFace repository to retrieve the tokenizer and chattemplate. An internet connection is required for the tokenizer initialization (only once), and you can use a repository different from that of the model as long as the tokenizer is compatible and the chattemplate meets your needs. Tokenizer gets downloaded for the first time in the models directory*

---

### **For Multimodal Encoder Model (.rknn) Installation**
1. **Download the encoder model .rknn**
- Download `.rknn` models directly from [Hugging Face](https://huggingface.co).
- Alternatively, convert your ONNX models into `.rknn` format.
- Place the `.rknn` model inside the `models` directory. RKLLama detected the encoder model present in the directory.
- Include manually the following properties in the `Modelfile` according to the conversion properties used for the conversion of the vision encoder `.rknn`:
```env
 IMAGE_WIDTH=448
 IMAGE_HEIGHT=
 N_IMAGE_TOKENS=
 IMG_START=
 IMG_END=
 IMG_CONTENT=

 # For example, for Qwen2VL/Qwen2.5VL:

 IMAGE_WIDTH=392
 IMAGE_HEIGHT=392
 N_IMAGE_TOKENS=196
 IMG_START=<|vision_start|>
 IMG_END=<|vision_end|>
 IMG_CONTENT=<|image_pad|>

 # For example, for MiniCPMV4:

 IMAGE_WIDTH=448
 IMAGE_HEIGHT=448
 N_IMAGE_TOKENS=64
 IMG_START=<image>
 IMG_END=</image>
 IMG_CONTENT=<unk>

Example directory structure for multimodal:

~/RKLLAMA/models/
    └── qwen2-vision\:2b
        |── Modelfile
        └── Qwen2-VL-2B-Instruct.rkllm
        └── Qwen2-VL-2B-Instruct.rknn

For Image Generation Installation

1. In a temporary folder, clone the repository https://huggingface.co/danielferr85/lcm-sd-1.5-rknn-2.3.2-rk3588 or https://huggingface