{"data":{"slug":"xorbitsai-inference","name":"inference","tagline":"Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready","github_url":"https://github.com/xorbitsai/inference","owner":"xorbitsai","repo":"inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/109655068?v=4","primary_language":"Python","stars":9423,"forks":846,"topics":["artificial-intelligence","chatglm","deployment","flan-t5","gemma","ggml","glm4","inference","llama","llama3","llamacpp","llm","machine-learning","mistral","openai-api","pytorch","qwen","vllm","whisper","wizardlm"],"archived":false,"github_pushed_at":"2026-07-11T16:10:51+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/xorbitsai-inference","markdown_url":"https://www.graphcanon.com/tools/xorbitsai-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xorbitsai-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xorbitsai-inference","description":"Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.","homepage_url":"https://inference.readthedocs.io","license":"Apache-2.0","open_issues":50,"watchers":60,"ai_summary":null,"readme_excerpt":"## Getting Started\n\n* [Docs](https://inference.readthedocs.io/en/latest/index.html)\n* [Built-in Models](https://inference.readthedocs.io/en/latest/models/builtin/index.html)\n* [Custom Models](https://inference.readthedocs.io/en/latest/models/custom.html)\n* [Deployment Docs](https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html)\n\n---\n\n### Docker \n\nNvidia GPU users can start Xinference server using [Xinference Docker Image](https://inference.readthedocs.io/en/latest/getting_started/using_docker_image.html). Prior to executing the installation command, ensure that both [Docker](https://docs.docker.com/get-docker/) and [CUDA](https://developer.nvidia.com/cuda-downloads) are set up on your system.\n\n```bash\ndocker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0\n```\n\n---\n\n# install xinference\nhelm install xinference xinference/xinference -n xinference --version 0.0.1-v<xinference_release_version>\n```\n\nFor more customized installation methods on K8s, please refer to the [documentation](https://inference.readthedocs.io/en/latest/getting_started/using_kubernetes.html).\n\n---\n\n### Quick Start\n\nInstall Xinference by using pip as follows. (For more options, see [Installation page](https://inference.readthedocs.io/en/latest/getting_started/installation.html).)\n\n```bash\npip install \"xinference[all]\"\n```\n\nTo start a local instance of Xinference, run the following command:\n\n```bash\n$ xinference-local\n```\n\nOnce Xinference is running, there are multiple ways you can try it: via the web UI, via cURL,\n via the command line, or via the Xinference’s python client. Check out our [docs]( https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html#run-xinference-locally) for the guide.","github_created_at":"2023-06-14T07:05:04+00:00","created_at":"2026-07-11T23:13:03.567739+00:00","updated_at":"2026-07-12T09:26:56.181262+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"artificial-intelligence","name":"artificial-intelligence"},{"slug":"chatglm","name":"chatglm"},{"slug":"deployment","name":"deployment"},{"slug":"flan-t5","name":"flan-t5"},{"slug":"gemma","name":"gemma"},{"slug":"ggml","name":"ggml"},{"slug":"glm4","name":"glm4"},{"slug":"inference","name":"inference"}],"trust":{"provenance":{"is_fork":false,"github_id":653496050,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:13:10.315Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":7,"days_since_push":0,"last_release_at":"2026-07-04T11:45:37Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:13:10.814Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:13:10.041Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T23:13:10.041Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:13:10.041Z"}}}}