{"data":{"slug":"xamey-deploy-llms-with-ansible","name":"deploy-llms-with-ansible","tagline":"Easily deploy LLMs using Ansible","github_url":"https://github.com/xamey/deploy-llms-with-ansible","owner":"xamey","repo":"deploy-llms-with-ansible","owner_avatar_url":"https://avatars.githubusercontent.com/u/34269296?v=4","primary_language":null,"stars":3,"forks":0,"topics":[],"archived":false,"github_pushed_at":"2025-05-01T21:58:25+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/xamey-deploy-llms-with-ansible","markdown_url":"https://www.graphcanon.com/tools/xamey-deploy-llms-with-ansible.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xamey-deploy-llms-with-ansible","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xamey-deploy-llms-with-ansible","description":"Easily deploy LLMs with Ansible. Uses Docker with llama.cpp or ollama. Secured with whitelisted IPs.","homepage_url":null,"license":null,"open_issues":0,"watchers":1,"ai_summary":"This repository facilitates the deployment of Large Language Models (LLMs) using Ansible scripts that can deploy either Ollama or llama.cpp on a Debian-based VM with Docker. It ensures security by allowing access only from whitelisted IPs.","readme_excerpt":"# Deploy LLMs with Ansible\n\nDeploy and serve LLMs using either Ollama or llama.cpp on a Debian-based VM.\nOnly whitelisted IPs will be able to request it.\n\n## Prerequisites\n\n- Ansible installed on your local machine\n- Debian-based VM with Docker installed\n- SSH access to the VM\n\n## Configuration\n\nModify the inventory files under `ollama` or `llamacpp` folders, depending on which technology you want to use.\n\nEach folder contains an `inventory.example.yml` file that you can use as a template. Copy the example file to create your own `inventory.yml`:\n\n```bash\n# For Ollama\ncp ollama/inventory.example.yml ollama/inventory.yml\n\n# For llama.cpp\ncp llamacpp/inventory.example.yml llamacpp/inventory.yml\n```\n\nThen modify the `inventory.yml` file with your specific configuration.\n\n### llama.cpp Configuration\n\nConfigure the following variables in your inventory file:\n\n- `ansible_host`: The IP address of your VM\n- `ansible_user`: The SSH username for connection\n- `ansible_ssh_private_key_file`: Path to your SSH private key file\n- `model_url`: URL to the GGUF model file\n- `allowed_ip_for_8080`: IP addresses allowed to access the server (port 8080)\n\n### Ollama Configuration\n\nConfigure the following variables in your inventory file:\n\n- `ansible_host`: The IP address of your VM\n- `ansible_user`: The SSH username for connection\n- `ansible_ssh_private_key_file`: Path to your SSH private key file\n- `model_name`: The model name as defined in Ollama library (e.g., for Qwen3-0.6B, use `qwen3:0.6b`)\n- `allowed_ip_for_8080`: IP addresses allowed to access the server (port 8080)\n\n## Deployment\n\n### Deploy with Ollama\n\n```bash\nansible-playbook -i ollama/inventory.yml ollama/playbook.yml\n```\n\n### Deploy with llama.cpp\n\n```bash\nansible-playbook -i llamacpp/inventory.yml llamacpp/playbook.yml\n```\n\n## API Usage\n\nThe server will be available on port 8080. You can make requests using the OpenAI API specification.\n\nExample request:\n\n```bash\ncurl --location 'http://YOUR_VM_IP:8080/v1/chat/completions' \\\n--header 'Content-Type: application/json' \\\n--header 'Authorization: Bearer no-key' \\\n--data '{\n    \"model\": \"your-model-name\",\n    \"messages\": [\n        {\n            \"role\": \"system\",\n            \"content\": \"You are a helpful assistant\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Hello, how are you?\"\n        }\n    ]\n}'\n```\n\n### Model Name Specification\n\nThe `model` field in your request must match exactly with the model you deployed:\n\n- for Ollama: use the same value as the `model_name` specified in your `inventory.yml` file\n- for llama.cpp: value isn't important","github_created_at":"2025-05-01T21:19:08+00:00","created_at":"2026-07-11T10:38:02.927363+00:00","updated_at":"2026-07-12T01:33:15.273137+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ansible","name":"ansible"},{"slug":"deployment","name":"deployment"},{"slug":"docker","name":"docker"},{"slug":"vm","name":"vm"},{"slug":"llama-cpp","name":"llama.cpp"},{"slug":"whitelisting","name":"whitelisting"},{"slug":"ollama","name":"ollama"}],"trust":{"provenance":{"is_fork":false,"github_id":976297111,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:38:03.462Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":435,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:38:04.389Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T01:33:15.209Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"unknown"},"requirements":{"notes":["Requires Ansible installed and configured on the local machine.","Debian-based VM with SSH access and Docker must be present."],"min_ram_gb":null,"requires_docker":true},"constraints":{"min_ram_gb":null,"pricing_model":"unknown","requires_docker":true},"when_to_use":["When you prefer using Ansible to automate the deployment of LLMs on a Debian-based virtual machine equipped with Docker.","If your environment already has familiarity and reliance on Ansible for configuration management tasks.","For scenarios where securing LLM access by whitelisting specific IPs is necessary."],"when_not_to_use":["When working in an environment that uses alternative automation tools like Terraform or Chef, as this tool specifically requires Ansible knowledge.","If the infrastructure does not support or permit the use of Docker for containerizing applications.","In cases where extensive customization of models beyond what llama.cpp and Ollama offer is required."],"source":"enrich:decision_facts","observed_at":"2026-07-12T01:31:22.692Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"unknown","requires_docker":true},"decision_summary":[{"label":"Pricing","value":"unknown"},{"label":"Requirements","value":"Requires Docker; Requires Ansible installed and configured on the local machine.; Debian-based VM with SSH access and Docker must be present."},{"label":"Adopt for","value":"deploy-llms-with-ansible"}]}}