{"data":{"slug":"autodeployai-ai-serving","name":"ai-serving","tagline":"Serving AI/ML models in open standard formats PMML and ONNX with HTTP and gRPC endpoints","github_url":"https://github.com/autodeployai/ai-serving","owner":"autodeployai","repo":"ai-serving","owner_avatar_url":"https://avatars.githubusercontent.com/u/50666721?v=4","primary_language":"Scala","stars":166,"forks":31,"topics":["ai-serving","inference","inference-server","onnx","onnx-grpc","onnx-inference","onnx-models","onnx-realtime","onnx-rest","pmml","pmml-deployment","pmml-grpc","pmml-inference","pmml-model","pmml-realtime","pmml-rest"],"archived":false,"github_pushed_at":"2026-02-24T02:56:29+00:00","maintenance_label":"Slowing","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/autodeployai-ai-serving","markdown_url":"https://www.graphcanon.com/tools/autodeployai-ai-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/autodeployai-ai-serving","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=autodeployai-ai-serving","description":"Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints","homepage_url":null,"license":"Apache-2.0","open_issues":3,"watchers":4,"ai_summary":"autodeployai/ai-serving is an inference server for both PMML and ONNX model formats supporting deployment via REST API or gRPC, deployable using Docker images or built from source requiring sbt build system.","readme_excerpt":"### Install using Docker\nThe easiest and most straight-forward way of using AI-Serving is with [Docker images](dockerfiles).\n\n---\n\n### Install from Source\n\n#### Install SBT\n\nThe [`sbt`](https://www.scala-sbt.org/) build system is required. After sbt installed, clone this repository, then change into the repository root directory:\n```bash\ncd REPO_ROOT\n```\n\n#### Build Assembly\n\nAI-Serving depends on [ONNX Runtime](https://github.com/microsoft/onnxruntime) to support ONNX models, and the default CPU accelerator (OpenMP) is used for ONNX Runtime:\n```bash\n\nsbt clean assembly\n```\n\nSet the property `-Dgpu=true` to use the GPU accelerator (CUDA) for [ONNX Runtime](https://github.com/microsoft/onnxruntime):\n```bash\nsbt -Dgpu=true clean assembly\n```\n\nRun `set test in assembly := {}` to disable unit tests if you want to skip them when generating an assembly jar:\n```bash\nsbt -Dgpu=true 'set test in assembly := {}' clean assembly\n```\n\nAn assembly jar will be generated:\n```bash\n$REPO_ROOT/target/scala-2.13/ai-serving-assembly-<version>.jar or ai-serving-gpu-assembly-<version>.jar\n```\n\n#### Start Server\n\nSimply run with the default CPU backend for ONNX models:\n```bash\njava -jar ai-serving-assembly-<version>.jar\n```\n\nGPU backend for ONNX models:\n```bash\njava -Donnxruntime.backend=cuda -jar ai-serving-gpu-assembly-<version>.jar\n```\nSeveral available execution backends: TensorRT, DirectML, Dnnl and so on. See [Advanced ONNX Runtime Configuration](#advanced-onnx-runtime-configuration) for details.\n\n#### Server Configurations\n\nBy default, the HTTP endpoint is listening on `http://0.0.0.0:9090/`, and the gRPC port is `9091`. You can customize those options that are defined in the [`application.conf`](src/main/resources/application.conf). There are several ways to override the default options, one is to create a new config file based on the default one, then:\n\n```bash\njava -Dconfig.file=/path/to/config-file -jar ai-serving-assembly-<version>.jar\n```\n\nAnother is to override each by setting Java system property, for example:\n\n```bash\njava -Dservice.http.port=9000 -Dservice.grpc.port=9001 -Dservice.home=\"/path/to/writable-directory\" -jar ai-serving-assembly-<version>.jar\n```\n\nAI-Serving is designed to be persistent or recoverable, so it needs a place to save all served models, that is specified by the property `service.home` that takes `/opt/ai-serving` as default, and the directory must be writable.\n\n---\n\n### Manual Deployment\n  To deploy a model manually, follow these steps:\n  1. Locate the directory specified by the `service.home` property.\n  2. Create a subdirectory named `models` if it does not already exist.\n  3. Inside the model name directory, create a subdirectory for the model version (for example, 1, 2, etc.).\n  4. Place the model file into the version directory:\n     - Use the fixed filename `model.pmml` for PMML models.\n     - Use the fixed filename `model.onnx` for ONNX models.\n  5. Customized inference behavior can be configured in model.conf, which can be placed either in the model directory or within a specific version directory. The following parameters are supported:\n     - max-batch-size=8 \n     \n       Specifies the maximum number of inference requests that can be grouped into a single batch. This option is effective only if the deployed ONNX model supports dynamic input shapes.\n     - max-batch-delay-ms=10 (milliseconds)\n\n       The maximum time the server waits to accumulate requests before forming a batch. If the batch does not reach max-batch-size within this time window, it will be executed with the available requests.\n     - request-timeout-ms=20 (milliseconds)\n\n       The maximum time allowed for processing an inference request. Requests exceeding this duration will be terminated or returned with a timeout error (HTTP: 504, gRPC: 4 DEADLINE_EXCEEDED). \n     - warmup-count=100\n\n       The number of warm-up inference runs executed when the model is loaded.\n     - warmup-data-type=zero (options: `zero`, `random`)\n\n       Speci","github_created_at":"2020-04-16T11:24:35+00:00","created_at":"2026-07-15T11:20:07.275316+00:00","updated_at":"2026-09-20T05:20:49.381803+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ai-serving","name":"ai-serving"},{"slug":"grpc","name":"grpc"},{"slug":"inference-server","name":"inference-server"},{"slug":"onnx","name":"onnx"},{"slug":"pmml-deployment","name":"pmml-deployment"},{"slug":"rest-api","name":"rest-api"}],"trust":{"provenance":{"is_fork":false,"github_id":256194101,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T05:20:47.594Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":208,"last_release_at":"2026-02-23T04:50:01Z","stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T11:20:11.027Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T05:20:48.646Z"},"languages":{"value":["scala"],"source":"github.language","observed_at":"2026-09-20T05:20:48.646Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T05:20:48.646Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to serve models in both PMML and ONNX formats without manual configuration changes between formats.","For projects requiring deployment of AI/ML models over REST API or gRPC interfaces for real-time inference scenarios.","If you want an option to deploy with Docker, simplifying setup across different environments.","When deploying on systems where a Scala environment is already part of the infrastructure stack."],"when_not_to_use":["Avoid if your team lacks familiarity or willingness to use Scala for deployment through sbt build system for customization needs.","Not suitable when only one model format, either PMML or ONNX but not both, is needed and a simpler solution would suffice.","If your project strictly requires a non-Dockerized setup that does not align with using pre-built Docker images."],"source":"enrich:decision_facts","observed_at":"2026-07-17T11:38:50.509Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Ai-Serving is an inference server supporting PMML and ONNX formats via HTTP or gRPC endpoints, easily deployable with Docker."}]}}