{"data":{"slug":"deep-diver-llm-as-chatbot","name":"LLM-As-Chatbot","tagline":"LLM as a Chatbot Service","github_url":"https://github.com/deep-diver/LLM-As-Chatbot","owner":"deep-diver","repo":"LLM-As-Chatbot","owner_avatar_url":"https://avatars.githubusercontent.com/u/26025527?v=4","primary_language":"Python","stars":3317,"forks":375,"topics":[],"archived":false,"github_pushed_at":"2023-11-20T14:33:58+00:00","maintenance_label":"Dormant","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/deep-diver-llm-as-chatbot","markdown_url":"https://www.graphcanon.com/tools/deep-diver-llm-as-chatbot.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/deep-diver-llm-as-chatbot","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=deep-diver-llm-as-chatbot","description":"LLM as a Chatbot Service","homepage_url":null,"license":"Apache-2.0","open_issues":19,"watchers":48,"ai_summary":"A repository for deploying various instruction-following fine-tuned LLM models as chatbot services, supporting Gradio and Discord bot interfaces, with internet search capabilities.","readme_excerpt":"## UPDATE\n- **Internet search support**: you can enable **internet search** capability in Gradio application and Discord bot. For gradio, there is a `internet mode` option in the control panel. For discord, you need to specify `--internet` option in your prompt. For both cases, you need a Serper API Key which you can get one from [serper.dev](https://serper.dev/). By signing up, you will get free 2,500 free google searches which is pretty much sufficient for a long-term test.\n- **Discord Bot support**: you can serve any model from the model zoo as Discord Bot. Find how to do this in the instruction section below.\n\n# 💬🚀 LLM as a Chatbot Service\n\nThe purpose of this repository is to let people to use lots of open sourced instruction-following fine-tuned LLM models as a Chatbot service. Because different models behave differently, and different models require differently formmated prompts, I made a very simple library [`Ping Pong`](https://github.com/deep-diver/PingPong) for model agnostic conversation and context managements. \n\nAlso, I made [`GradioChat`](https://github.com/deep-diver/gradio-chat) UI that has a similar shape to [HuggingChat](https://huggingface.co/chat/) but entirely built in Gradio. Those two projects are fully integrated to power this project. \n\n## Easiest way to try out ( ✅ Gradio, 🚧 Discord Bot )\n\n### Jarvislabs.ai\n\nThis project has become the one of the default framework at [jarvislabs.ai](https://jarvislabs.ai/). Jarvislabs.ai is one of the cloud GPU VM provider with the cheapest GPU prices. Furthermore, all the weights of the supported popular open source LLMs are pre-downloaded. You don't need to waste of your money and time to wait until download hundreds of GBs to try out a collection of LLMs. In less than 10 minutes, you can try out any model. \n- for further instruction how to run Gradio application, please follow the [official documentation](https://jarvislabs.ai/docs/llmchat) on the `llmchat` framework.\n\n### dstack\n\n[`dstack`](https://dstack.ai) is an open-source tool that allows to run LLM-based apps in a a cloud of your choice via single command. `dstack` supports AWS, GCP, Azure, Lambda Cloud, etc.\n\nUse the `gradio.dstack.yml` and `discord.dstack.yml` configurations to run the Gradio app and Discord bot via `dstack`.\n- for more details on how to run this repo with `dstack`, read the [official documentation](https://dstack.ai/examples/llmchat) by `dstack`.\n\n## Instructions\n\n### Standalone Gradio app\n\n\n\n0. Prerequisites\n\n    Note that the code only works `Python >= 3.9` and `gradio >= 3.32.0`\n\n    ```console\n    $ conda create -n llm-serve python=3.9\n    $ conda activate llm-serve\n    ```\n\n1. Install dependencies. \n    ```console\n    $ cd LLM-As-Chatbot\n    $ pip install -r requirements.txt\n    ```\n\n2. Run Gradio application\n\n    There is no required parameter to run the Gradio application. However, there are some small details worth being noted. When `--local-files-only` is set, application won't try to look up the Hugging Face Hub(remote). Instead, it will only use the files already downloaded and cached.\n\n    Hugging Face libraries stores downloaded contents under `~/.cache` by default, and this application assumes so. However, if you downloaded weights in different location for some reasons, you can set `HF_HOME` environment variable. Find more about the [environment variables here](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables)\n\n   In order to leverage **internet search** capability, you need Serper API Key. You can set it manually in the control panel or in CLI. When specifying the Serper API Key in CLI, it will be injected into the corresponding UI control. If you don't have it yet, please get one from [serper.dev](https://serper.dev/). By signing up, you will get free 2,500 free google searches which is pretty much sufficient for a long-term test.\n\n    ```console\n    $ python app.py --root-path \"\" \\\n                    --local-files-only \\","github_created_at":"2023-02-27T15:32:16+00:00","created_at":"2026-07-11T23:10:18.50469+00:00","updated_at":"2026-09-20T04:29:31.668112+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"chatbot","name":"chatbot"},{"slug":"cloud-gpu","name":"cloud gpu"},{"slug":"context-management","name":"context management"},{"slug":"conversation-management","name":"conversation management"},{"slug":"discord","name":"discord"},{"slug":"gradio","name":"gradio"},{"slug":"internet-search","name":"internet search"},{"slug":"model-agnostic","name":"model agnostic"}],"trust":{"provenance":{"is_fork":false,"github_id":607233462,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:16:29.008Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1032,"last_release_at":null,"stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":9,"high_count":0,"last_scan_at":"2026-09-18T08:16:31.219Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:16:27.919Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:16:27.919Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:16:27.919Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to deploy a variety of LLM models as chatbots with internet search capabilities, using either Gradio or Discord bot interfaces.","If you are working with cloud GPU VM providers like Jarvislabs.ai, where the weights of popular open-source LLMs are pre-downloaded, saving time and resources.","When you want to leverage the `Ping Pong` library for model-agnostic conversation and context management, and the `GradioChat` UI for a user-friendly chatbot interface."],"when_not_to_use":["If you require a chatbot service that does not support internet search capabilities, as LLM-As-Chatbot mandates the use of a Serper API Key for enabling this feature.","When you need a chatbot service that does not integrate with Gradio or Discord, as LLM-As-Chatbot is specifically designed for these interfaces.","If you are working with Python versions less than 3.9, as the framework requires Python 3.9 or higher."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:41:51.318Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"LLM-As-Chatbot is a Python-based framework for deploying instruction-following fine-tuned LLM models as chatbot services, with support for Gradio and Discord bot interfaces, and internet search capabilities."}]}}