{"data":{"slug":"autodeployai-ai-serving","name":"ai-serving","tagline":"Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints","github_url":"https://github.com/autodeployai/ai-serving","owner":"autodeployai","repo":"ai-serving","owner_avatar_url":"https://avatars.githubusercontent.com/u/50666721?v=4","primary_language":"Scala","stars":166,"forks":31,"topics":["ai-serving","inference","inference-server","onnx","onnx-grpc","onnx-inference","onnx-models","onnx-realtime","onnx-rest","pmml","pmml-deployment","pmml-grpc","pmml-inference","pmml-model","pmml-realtime","pmml-rest"],"archived":false,"github_pushed_at":"2026-02-24T02:56:29+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/autodeployai-ai-serving","markdown_url":"https://www.graphcanon.com/tools/autodeployai-ai-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/autodeployai-ai-serving","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=autodeployai-ai-serving","description":"Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints","homepage_url":null,"license":"Apache-2.0","open_issues":3,"watchers":4,"ai_summary":null,"readme_excerpt":"### Install using Docker\nThe easiest and most straight-forward way of using AI-Serving is with [Docker images](dockerfiles).\n\n---\n\n### Install from Source\n\n#### Install SBT\n\nThe [`sbt`](https://www.scala-sbt.org/) build system is required. After sbt installed, clone this repository, then change into the repository root directory:\n```bash\ncd REPO_ROOT\n```\n\n#### Build Assembly\n\nAI-Serving depends on [ONNX Runtime](https://github.com/microsoft/onnxruntime) to support ONNX models, and the default CPU accelerator (OpenMP) is used for ONNX Runtime:\n```bash\n\nsbt clean assembly\n```\n\nSet the property `-Dgpu=true` to use the GPU accelerator (CUDA) for [ONNX Runtime](https://github.com/microsoft/onnxruntime):\n```bash\nsbt -Dgpu=true clean assembly\n```\n\nRun `set test in assembly := {}` to disable unit tests if you want to skip them when generating an assembly jar:\n```bash\nsbt -Dgpu=true 'set test in assembly := {}' clean assembly\n```\n\nAn assembly jar will be generated:\n```bash\n$REPO_ROOT/target/scala-2.13/ai-serving-assembly-<version>.jar or ai-serving-gpu-assembly-<version>.jar\n```\n\n#### Start Server\n\nSimply run with the default CPU backend for ONNX models:\n```bash\njava -jar ai-serving-assembly-<version>.jar\n```\n\nGPU backend for ONNX models:\n```bash\njava -Donnxruntime.backend=cuda -jar ai-serving-gpu-assembly-<version>.jar\n```\nSeveral available execution backends: TensorRT, DirectML, Dnnl and so on. See [Advanced ONNX Runtime Configuration](#advanced-onnx-runtime-configuration) for details.\n\n#### Server Configurations\n\nBy default, the HTTP endpoint is listening on `http://0.0.0.0:9090/`, and the gRPC port is `9091`. You can customize those options that are defined in the [`application.conf`](src/main/resources/application.conf). There are several ways to override the default options, one is to create a new config file based on the default one, then:\n\n```bash\njava -Dconfig.file=/path/to/config-file -jar ai-serving-assembly-<version>.jar\n```\n\nAnother is to override each by setting Java system property, for example:\n\n```bash\njava -Dservice.http.port=9000 -Dservice.grpc.port=9001 -Dservice.home=\"/path/to/writable-directory\" -jar ai-serving-assembly-<version>.jar\n```\n\nAI-Serving is designed to be persistent or recoverable, so it needs a place to save all served models, that is specified by the property `service.home` that takes `/opt/ai-serving` as default, and the directory must be writable.\n\n---\n\n### Manual Deployment\n  To deploy a model manually, follow these steps:\n  1. Locate the directory specified by the `service.home` property.\n  2. Create a subdirectory named `models` if it does not already exist.\n  3. Inside the model name directory, create a subdirectory for the model version (for example, 1, 2, etc.).\n  4. Place the model file into the version directory:\n     - Use the fixed filename `model.pmml` for PMML models.\n     - Use the fixed filename `model.onnx` for ONNX models.\n  5. Customized inference behavior can be configured in model.conf, which can be placed either in the model directory or within a specific version directory. The following parameters are supported:\n     - max-batch-size=8 \n     \n       Specifies the maximum number of inference requests that can be grouped into a single batch. This option is effective only if the deployed ONNX model supports dynamic input shapes.\n     - max-batch-delay-ms=10 (milliseconds)\n\n       The maximum time the server waits to accumulate requests before forming a batch. If the batch does not reach max-batch-size within this time window, it will be executed with the available requests.\n     - request-timeout-ms=20 (milliseconds)\n\n       The maximum time allowed for processing an inference request. Requests exceeding this duration will be terminated or returned with a timeout error (HTTP: 504, gRPC: 4 DEADLINE_EXCEEDED). \n     - warmup-count=100\n\n       The number of warm-up inference runs executed when the model is loaded.\n     - warmup-data-type=zero (options: `zero`, `random`)\n\n       Speci","github_created_at":"2020-04-16T11:24:35+00:00","created_at":"2026-07-15T11:20:07.275316+00:00","updated_at":"2026-07-15T11:20:13.803256+00:00","categories":[{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ai-serving","name":"ai-serving"},{"slug":"inference","name":"inference"},{"slug":"inference-server","name":"inference-server"},{"slug":"onnx","name":"onnx"},{"slug":"onnx-grpc","name":"onnx-grpc"},{"slug":"onnx-inference","name":"onnx-inference"},{"slug":"onnx-models","name":"onnx-models"},{"slug":"onnx-realtime","name":"onnx-realtime"}],"trust":{"provenance":{"is_fork":false,"github_id":256194101,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T11:20:09.899Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":141,"last_release_at":"2026-02-23T04:50:01Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T11:20:11.027Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T11:20:09.392Z"},"languages":{"value":["scala"],"source":"github.language","observed_at":"2026-07-15T11:20:09.392Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-15T11:20:09.392Z"}}}}