{"data":{"slug":"ray-project-llm-applications","name":"llm-applications","tagline":"Comprehensive guide to building RAG-based LLM applications for production","github_url":"https://github.com/ray-project/llm-applications","owner":"ray-project","repo":"llm-applications","owner_avatar_url":"https://avatars.githubusercontent.com/u/22125274?v=4","primary_language":"Jupyter Notebook","stars":1855,"forks":256,"topics":["anyscale","fine-tuning","llama2","llms","machine-learning","openai","ray","serving"],"archived":false,"github_pushed_at":"2026-08-15T00:14:04+00:00","maintenance_label":"Active","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/ray-project-llm-applications","markdown_url":"https://www.graphcanon.com/tools/ray-project-llm-applications.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ray-project-llm-applications","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ray-project-llm-applications","description":"A comprehensive guide to building RAG-based LLM applications for production.","homepage_url":null,"license":"CC-BY-4.0","open_issues":13,"watchers":16,"ai_summary":"Offers instructions and resources focussed on developing large language model applications using Ray for production environments.","readme_excerpt":"# LLM Applications\n\nA comprehensive guide to building RAG-based LLM applications for production.\n\n- **Blog post**: https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1\n- **GitHub repository**: https://github.com/ray-project/llm-applications\n- **Interactive notebook**: https://github.com/ray-project/llm-applications/blob/main/notebooks/rag.ipynb\n- **Anyscale Endpoints**: https://endpoints.anyscale.com/\n- **Ray documentation**: https://docs.ray.io/\n\nIn this guide, we will learn how to:\n\n- 💻 Develop a retrieval augmented generation (RAG) based LLM application from scratch.\n- 🚀 Scale the major components (load, chunk, embed, index, serve, etc.) in our application.\n- ✅ Evaluate different configurations of our application to optimize for both per-component (ex. retrieval_score) and overall performance (quality_score).\n- 🔀 Implement LLM hybrid routing approach to bridge the gap b/w OSS and closed LLMs.\n- 📦 Serve the application in a highly scalable and available manner.\n- 💥 Share the 1st order and 2nd order impacts LLM applications have had on our products.\n\n
\n\n\n## Setup\n\n### API keys\nWe'll be using [OpenAI](https://platform.openai.com/docs/models/) to access ChatGPT models like `gpt-3.5-turbo`, `gpt-4`, etc. and [Anyscale Endpoints](https://endpoints.anyscale.com/) to access OSS LLMs like `Llama-2-70b`. Be sure to create your accounts for both and have your credentials ready.\n\n### Compute\n
\n Local\n You could run this on your local laptop but a we highly recommend using a setup with access to GPUs. You can set this up on your own or on [Anyscale](http://anyscale.com/).\n
\n\n
\n Anyscale
\n\n\n
\n\n### Repository\n```bash\ngit clone https://github.com/ray-project/llm-applications.git .\ngit config --global user.name \ngit config --global user.email \n```\n\n### Data\nOur data is already ready at `/efs/shared_storage/goku/docs.ray.io/en/master/` (on Staging, `us-east-1`) but if you wanted to load it yourself, run this bash command (change `/desired/output/directory`, but make sure it's on the shared storage,\nso that it's accessible to the workers)\n```bash\ngit clone https://github.com/ray-project/llm-applications.git .\n```\n\n### Environment\n\nThen set up the environment correctly by specifying the values in your `.env` file,\nand installing the dependencies:\n\n```bash\npip install --user -r requirements.txt\nexport PYTHONPATH=$PYTHONPATH:$PWD\npre-commit install\npre-commit autoupdate\n```\n\n### Credentials\n```bash\ntouch .env\n# Add environment variables to .env\nOPENAI_API_BASE=\"https://api.openai.com/v1\"\nOPENAI_API_KEY=\"\" # https://platform.openai.com/account/api-keys\nANYSCALE_API_BASE=\"https://api.endpoints.anyscale.com/v1\"\nANYSCALE_API_KEY=\"\" # https://app.endpoints.anyscale.com/credentials\nDB_CONNECTION_STRING=\"dbname=postgres user=postgres host=localhost password=postgres\"\nsource .env\n```\n\nNow we're ready to go through the [rag.ipynb](notebooks/rag.ipynb) interactive notebook to develop and serve our LLM application!\n\n### Learn more\n- If y","github_created_at":"2023-08-16T04:41:26+00:00","created_at":"2026-07-11T11:38:47.593672+00:00","updated_at":"2026-08-24T00:02:03.849757+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"anyscale","name":"anyscale"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"llama2","name":"llama2"},{"slug":"machin-learning","name":"machin-learning"},{"slug":"openai","name":"openai"},{"slug":"ray","name":"ray"},{"slug":"serving","name":"serving"}],"trust":{"provenance":{"is_fork":false,"github_id":679091311,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-24T00:02:03.109Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":8,"last_release_at":"2023-09-17T19:22:49Z","stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:38:48.656Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-24T00:02:03.560Z"},"languages":{"value":["jupyter notebook","python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-24T00:02:03.560Z"},"license_spdx":{"value":"CC-BY-4.0","source":"github.license","observed_at":"2026-08-24T00:02:03.560Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You require a detailed guide specifically tailored to the development and deployment of RAG-based applications, leveraging Ray for performance and scalability.","Your team has experience or interest in working with Jupyter Notebooks as the primary format for tutorials and documentation."],"when_not_to_use":["If you are looking for a more generalized approach to LLM application development that does not specifically cater to RAG-based designs and Ray optimizations.","When your project workflow is incompatible with or cannot support Jupyter Notebook dependencies and the resources assume."],"source":"enrich:decision_facts","observed_at":"2026-07-16T23:10:38.521Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"The llm-applications repository offers focused guidance on deploying RAG-based LLM apps in production environments with an emphasis on using Ray."}]}}