{"data":{"slug":"xorbitsai-inference","name":"inference","tagline":"Unified production-ready inference API for various LLMs and models","github_url":"https://github.com/xorbitsai/inference","owner":"xorbitsai","repo":"inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/109655068?v=4","primary_language":"Python","stars":9575,"forks":866,"topics":["artificial-intelligence","deployment","diffusers","gemma","glm","glm-5-3","inference","kimi","kimi-k3","llama-cpp","llamacpp","llm","machine-learning","openai-api","pytorch","qwen","sglang","transformers","vllm","whisper"],"archived":false,"github_pushed_at":"2026-09-18T06:04:24+00:00","maintenance_label":"Very active","stars_delta_30d":105,"url":"https://www.graphcanon.com/tools/xorbitsai-inference","markdown_url":"https://www.graphcanon.com/tools/xorbitsai-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xorbitsai-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xorbitsai-inference","description":"Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.","homepage_url":"https://inference.readthedocs.io","license":"Apache-2.0","open_issues":39,"watchers":64,"ai_summary":"Xinference provides a single API for deploying and running a variety of open-source, speech, and multimodal models, including LLMs, on different environments like cloud, on-prem, or local machines.","readme_excerpt":"## Getting Started\n\n* [Docs](https://inference.readthedocs.io/en/latest/index.html)\n* [Built-in Models](https://inference.readthedocs.io/en/latest/models/builtin/index.html)\n* [Custom Models](https://inference.readthedocs.io/en/latest/models/custom.html)\n* [Deployment Docs](https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html)\n\n---\n\n### Docker \n\nNvidia GPU users can start Xinference server using [Xinference Docker Image](https://inference.readthedocs.io/en/latest/getting_started/using_docker_image.html). Prior to executing the installation command, ensure that both [Docker](https://docs.docker.com/get-docker/) and [CUDA](https://developer.nvidia.com/cuda-downloads) are set up on your system.\n\n```bash\ndocker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0\n```\n\n---\n\n# install xinference\nhelm install xinference xinference/xinference -n xinference --version 0.0.1-v<xinference_release_version>\n```\n\nFor more customized installation methods on K8s, please refer to the [documentation](https://inference.readthedocs.io/en/latest/getting_started/using_kubernetes.html).\n\n---\n\n### Quick Start\n\nInstall Xinference by using pip as follows. (For more options, see [Installation page](https://inference.readthedocs.io/en/latest/getting_started/installation.html).)\n\n```bash\npip install \"xinference[all]\"\n```\n\nTo start a local instance of Xinference, run the following command:\n\n```bash\n$ xinference-local\n```\n\nOnce Xinference is running, there are multiple ways you can try it: via the web UI, via cURL,\n via the command line, or via the Xinference’s python client. Check out our [docs]( https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html#run-xinference-locally) for the guide.","github_created_at":"2023-06-14T07:05:04+00:00","created_at":"2026-07-11T23:13:03.567739+00:00","updated_at":"2026-09-20T04:30:04.384248+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"artificial-intelligence","name":"artificial-intelligence"},{"slug":"deployment","name":"deployment"},{"slug":"diffusers","name":"diffusers"},{"slug":"gemma","name":"gemma"},{"slug":"glm","name":"glm"},{"slug":"glm-5-3","name":"glm-5-3"},{"slug":"kimi","name":"kimi"},{"slug":"kimi-k3","name":"kimi-k3"}],"trust":{"provenance":{"is_fork":false,"github_id":653496050,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:20:54.204Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":7,"days_since_push":0,"last_release_at":"2026-09-11T16:37:55Z","stars_delta_30d":105,"open_issues_delta_30d":-3},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:20:56.410Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:20:52.624Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-18T08:20:52.624Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:20:52.624Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:20:52.624Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The tool is open-source under the Apache-2.0 license, but additional services or support might incur costs."},"requirements":{"notes":["Docker and CUDA setup are required for Nvidia GPU users.","For Kubernetes deployment, Helm is required."],"min_ram_gb":4,"requires_docker":true},"constraints":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"when_to_use":["When you need a single API to deploy and run a variety of models, including LLMs, on cloud, on-premises, or local machines.","If you require flexibility to switch between different LLMs by changing a single line of code.","For users who want to deploy models using Docker, Kubernetes, or a local setup, with comprehensive documentation for each method."],"when_not_to_use":["If you are looking for a tool that supports only a specific type of model or environment, as Xinference is designed to be versatile across multiple models and deployment environments.","When you need a tool that does not require setting up Docker or Kubernetes for deployment, as Xinference provides detailed instructions for these setups but may not be suitable for users preferring a"],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:50:10.528Z"},"constraint_facets":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"decision_summary":[{"label":"Pricing","value":"freemium - The tool is open-source under the Apache-2.0 license, but additional services or support might incur costs."},{"label":"Requirements","value":"Min 4 GB RAM; Requires Docker; Docker and CUDA setup are required for Nvidia GPU users.; For Kubernetes deployment, Helm is required."},{"label":"Adopt for","value":"Xinference is a unified production-ready inference API for deploying and running various models, including LLMs, on different environments."}]}}