--- title: "chatWeb" type: "tool" slug: "skywalkerdarren-chatweb" canonical_url: "https://www.graphcanon.com/tools/skywalkerdarren-chatweb" github_url: "https://github.com/SkywalkerDarren/chatWeb" homepage_url: null stars: 913 forks: 137 primary_language: "Python" license: "MIT" categories: ["data-retrieval", "inference-serving"] tags: ["vector-database", "pdf", "ai", "crawler", "chatgpt", "embedding", "docx"] updated_at: "2026-07-07T18:48:58.018487+00:00" --- # chatWeb > Web crawling, text extraction from PDF/DOCX/TXT files, summarization, and Q&A based on GPT3.5 embedding API ChatWeb is a Python-based repository that allows users to crawl web pages, extract text content from various file types, and perform summarization or answer questions using GPT-3.5's embedding API with support for vector databases. ## Facts - Repository: https://github.com/SkywalkerDarren/chatWeb - Stars: 913 Β· Forks: 137 Β· Open issues: 7 Β· Watchers: 18 - Primary language: Python - License: MIT - Last pushed: 2026-05-25T16:56:25+00:00 ## Categories - [Data & Retrieval](/categories/data-retrieval.md) - [Inference & Serving](/categories/inference-serving.md) ## Tags vector-database, pdf, ai, crawler, chatgpt, embedding, docx ## Related tools - [transformers](/tools/huggingface-transformers.md) - πŸ€— Transformers: the model-definition framework for state-of-the-art machine learning models (β˜… 162,347) - [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (β˜… 151,298) - [firecrawl](/tools/firecrawl-firecrawl.md) - The API to search, scrape, and interact with the web at scale. (β˜… 147,117) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (β˜… 144,575) - [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (β˜… 85,612) - [PaddleOCR](/tools/paddlepaddle-paddleocr.md) - PaddleOCR: A powerful OCR toolkit for transforming PDFs/images into structured data (β˜… 84,919) - [graphify](/tools/graphify-labs-graphify.md) - AI coding assistant skill that transforms various file types into a queryable knowledge graph (β˜… 79,371) - [rtk](/tools/rtk-ai-rtk.md) - CLI proxy reducing LLM token consumption by 60-90% (β˜… 69,253) ## README (excerpt) ```text # ChatWeb [English Doc](readme.md) [δΈ­ζ–‡ζ–‡ζ‘£](readme.zh.md) ChatWeb can crawl any webpage or extract text from PDF, DOCX, TXT files, and generate an embedded summary. It can also answer your questions based on the content of the text. It is implemented using the chatAPI and embeddingAPI based on gpt3.5, as well as a vector database. # Basic Principle The basic principle is similar to existing projects such as chatPDF and automated customer service AI. Crawl web pages Extract text content Use GPT3.5's embedding API to generate vectors for each paragraph Calculate the similarity score between each paragraph's vector and the entire text's vector to generate a summary Store the vector-text mapping in a vector database Generate keywords from user input Generate a vector from the keywords Use the vector database to perform a nearest neighbor search and return a list of the most similar texts Use GPT3.5's chat API to design a prompt that answers the user's question based on the most similar texts in the list. The idea is to extract relevant content from a large amount of text and then answer questions based on that content, which can achieve a similar effect to breaking through token limits. An improvement was made to generate vectors based on keywords rather than the user's question, which increases the accuracy of searching for relevant texts. # Getting Started ## Manual installation: - Install Python3 - Download this repository by running `git clone https://github.com/SkywalkerDarren/chatWeb.git` - Navigate to the directory by running `cd chatWeb` - Copy `config.example.json` to `config.json` - Edit `config.json` and set `open_ai_key` to your OpenAI API key - Install dependencies by running `pip3 install -r requirements.txt` - Start the application by running `python3 main.py` ## Docker: if you prefer, you can also run this project using docker: - build the container using `docker-compose build` (only needed once when you are not planning to contibute to this repo) - copy `config.example.json` to `config.json` and set all the needed stuff. The example config is already fine for running with docker, no need to change anything there, if you don't have the OPEN_AI_KEY in your env variables you can set it here too, or later if you run this app. - run the container: `docker-compose up" - open the application in browser: `http://localhost:7860` ## Set language - Edit `config.json`, set `language` to `English` or other language ## Mode Selection - Edit `config.json` and set `mode` to `console`, `api`, or `webui` to choose the startup mode. - In `console` mode, type `/help` to view commands. - In `api` mode, an API service can be provided to the outside world. `api_port` and `api_host` can be set in `config.json`. - In `webui` mode, a web user interface service can be provided. `webui_port` can be set in `config.json`, defaulting to `http://127.0.0.1:7860`. ## Stream Mode - Edit `config.json` and set `use_stream` to `true`. ## Setting the Temperature - Edit `config.json` and set `temperature` to a value between 0 and 1. - The smaller the value, the more conservative and stable the response will be. The larger the value, the more daring the response may be, possibly resulting in "hallucinations." ## OpenAI Proxy Settings - Edit `config.json` and add `open_ai_proxy` for your proxy address, for example: ``` "open_ai_proxy": { "http": "socks5://127.0.0.1:1081", "https": "socks5://127.0.0.1:1081" } ``` ## Install PostgreSQL (Optional) - Edit `config.json` and set `use_postgres` to `true`. - Install PostgreSQL. - The default SQL address is `postgresql://localhost:5432/mydb`, or you can set it in `config.json`. - Install the pgvector plugin. Compile and install the extension (support Postgres 11+). ```bash git clone --branch v0.4.0 https://github.com/pgvector/pgvector.git cd pgvector make make install # may need sudo ``` Then load it in the database you want to use it in ```postgresql CREATE EXTENSION vector; ``` - ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/skywalkerdarren-chatweb`](/api/graphcanon/tools/skywalkerdarren-chatweb) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_