Requirements

• ONNX Runtime
• Boost
• CMake, pkg-config
• CUDA(optional, for Nvidia GPU support)
• OpenSSL(optional, for HTTPS)

Install ONNX Runtime

Linux

• Use download-onnxruntime-linux.sh script
  • This script downloads the latest version of the binary and install to /usr/local/onnxruntime.
  • Also, add /usr/local/onnxruntime/lib to /etc/ld.so.conf.d/onnxruntime.conf and run ldconfig.
• Or manually download binary from ONNX Runtime Releases.

Mac OS

brew install onnxruntime

Install dependencies

Ubuntu/Debian

sudo apt install cmake pkg-config libboost-all-dev libssl-dev

---

# optional, for Nvidia GPU support with Docker
sudo apt install nvidia-container-toolkit

Mac OS

brew install cmake boost openssl

Compile and Install

cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
sudo cmake --install build --prefix /usr/local/onnxruntime-server

Install via a package manager

Docker

• Docker hub: kibaes/onnxruntime-server
  • 1.27.1-linux-cuda13 amd64(CUDA 13.x, cuDNN 9.x)
  • 1.27.1-linux-cuda12 amd64(CUDA 12.x, cuDNN 9.x)
  • 1.27.1-linux-cpu amd64, arm64

DOCKER_IMAGE=kibaes/onnxruntime-server:1.27.1-linux-cuda13 # or 1.27.1-linux-cuda12 or 1.27.1-linux-cpu

docker pull ${DOCKER_IMAGE}