ai-serving

Install using Docker

The easiest and most straight-forward way of using AI-Serving is with Docker images.

Install from Source

Install SBT

The sbt build system is required. After sbt installed, clone this repository, then change into the repository root directory:

cd REPO_ROOT

Build Assembly

AI-Serving depends on ONNX Runtime to support ONNX models, and the default CPU accelerator (OpenMP) is used for ONNX Runtime:

sbt clean assembly

Set the property -Dgpu=true to use the GPU accelerator (CUDA) for ONNX Runtime:

sbt -Dgpu=true clean assembly

Run set test in assembly := {} to disable unit tests if you want to skip them when generating an assembly jar:

sbt -Dgpu=true 'set test in assembly := {}' clean assembly

An assembly jar will be generated:

$REPO_ROOT/target/scala-2.13/ai-serving-assembly-<version>.jar or ai-serving-gpu-assembly-<version>.jar

Start Server

Simply run with the default CPU backend for ONNX models:

java -jar ai-serving-assembly-<version>.jar

GPU backend for ONNX models:

java -Donnxruntime.backend=cuda -jar ai-serving-gpu-assembly-<version>.jar

Several available execution backends: TensorRT, DirectML, Dnnl and so on. See Advanced ONNX Runtime Configuration for details.

Server Configurations

By default, the HTTP endpoint is listening on http://0.0.0.0:9090/, and the gRPC port is 9091. You can customize those options that are defined in the application.conf. There are several ways to override the default options, one is to create a new config file based on the default one, then:

java -Dconfig.file=/path/to/config-file -jar ai-serving-assembly-<version>.jar

Another is to override each by setting Java system property, for example:

java -Dservice.http.port=9000 -Dservice.grpc.port=9001 -Dservice.home="/path/to/writable-directory" -jar ai-serving-assembly-<version>.jar

AI-Serving is designed to be persistent or recoverable, so it needs a place to save all served models, that is specified by the property service.home that takes /opt/ai-serving as default, and the directory must be writable.

Manual Deployment

To deploy a model manually, follow these steps:

1. Locate the directory specified by the service.home property.
2. Create a subdirectory named models if it does not already exist.
3. Inside the model name directory, create a subdirectory for the model version (for example, 1, 2, etc.).
4. Place the model file into the version directory:
   • Use the fixed filename model.pmml for PMML models.
   • Use the fixed filename model.onnx for ONNX models.
5. Customized inference behavior can be configured in model.conf, which can be placed either in the model directory or within a specific version directory. The following parameters are supported:

   • max-batch-size=8

     Specifies the maximum number of inference requests that can be grouped into a single batch. This option is effective only if the deployed ONNX model supports dynamic input shapes.

   • max-batch-delay-ms=10 (milliseconds)

     The maximum time the server waits to accumulate requests before forming a batch. If the batch does not reach max-batch-size within this time window, it will be executed with the available requests.

   • request-timeout-ms=20 (milliseconds)

     The maximum time allowed for processing an inference request. Requests exceeding this duration will be terminated or returned with a timeout error (HTTP: 504, gRPC: 4 DEADLINE_EXCEEDED).

   • warmup-count=100

     The number of warm-up inference runs executed when the model is loaded.

   • warmup-data-type=zero (options: zero, random)

     Speci