{"data":{"slug":"skywalkerdarren-chatweb","name":"chatWeb","tagline":"ChatWeb can crawl web pages and various document types for content extraction and summarization.","github_url":"https://github.com/SkywalkerDarren/chatWeb","owner":"SkywalkerDarren","repo":"chatWeb","owner_avatar_url":"https://avatars.githubusercontent.com/u/20706299?v=4","primary_language":"Python","stars":916,"forks":137,"topics":["ai","chatgpt","crawler","docx","embedding","faiss","gpt","gpt-35-turbo","news-extractor","newspaper","openai","pdf","pgvector","postgresql","vector-database"],"archived":false,"github_pushed_at":"2026-05-25T16:56:25+00:00","maintenance_label":"Steady","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/skywalkerdarren-chatweb","markdown_url":"https://www.graphcanon.com/tools/skywalkerdarren-chatweb.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/skywalkerdarren-chatweb","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=skywalkerdarren-chatweb","description":"ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.","homepage_url":null,"license":"MIT","open_issues":7,"watchers":18,"ai_summary":"A tool that crawls the web and processes documents to extract main content and summarize key points in response to queries.","readme_excerpt":"## Manual installation:\n\n- Install Python3\n- Download this repository by running `git clone https://github.com/SkywalkerDarren/chatWeb.git`\n- Navigate to the directory by running `cd chatWeb`\n- Copy `config.example.json` to `config.json`\n- Edit `config.json` and set `open_ai_key` to your OpenAI API key\n- Install dependencies by running `pip3 install -r requirements.txt`\n- Start the application by running `python3 main.py`\n\n---\n\n## Docker:\nif you prefer, you can also run this project using docker:\n\n- build the container using `docker-compose build` (only needed once when you are not planning to contibute to this repo)\n- copy `config.example.json` to `config.json` and set all the needed stuff. The example config is already fine for running with docker, no need to change anything there, if you don't have the OPEN_AI_KEY in your env variables you can set it here too, or later if you run this app.\n- run the container: `docker-compose up\"\n- open the application in browser: `http://localhost:7860`\n\n---\n\n## Install PostgreSQL (Optional)\n\n- Edit `config.json` and set `use_postgres` to `true`.\n- Install PostgreSQL.\n  - The default SQL address is `postgresql://localhost:5432/mydb`, or you can set it in `config.json`.\n- Install the pgvector plugin.\n\nCompile and install the extension (support Postgres 11+).\n\n```bash\ngit clone --branch v0.4.0 https://github.com/pgvector/pgvector.git\ncd pgvector\nmake\nmake install # may need sudo\n```\nThen load it in the database you want to use it in\n\n```postgresql\nCREATE EXTENSION vector;\n```\n\n- Install dependency with pip: `pip3 install psycopg2`","github_created_at":"2023-03-09T12:15:07+00:00","created_at":"2026-07-07T17:44:51.984514+00:00","updated_at":"2026-08-21T18:02:14.146406+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"chatgpt","name":"chatgpt"},{"slug":"crawler","name":"crawler"},{"slug":"docx","name":"docx"},{"slug":"embedding","name":"embedding"},{"slug":"faiss","name":"faiss"},{"slug":"gpt","name":"gpt"},{"slug":"news-extractor","name":"news-extractor"}],"trust":{"provenance":{"is_fork":false,"github_id":611723171,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-21T18:02:13.311Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":88,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:26:52.642Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-21T18:02:13.808Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-21T18:02:13.808Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-21T18:02:13.808Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-21T18:02:13.808Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-21T18:02:13.808Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You need to rapidly extract actionable insights or summaries from multiple sources of data including websites and documents.","Your goal is to integrate a conversational interface with content retrieval capabilities leveraging AI for natural language processing tasks.","The project requires the option to store embeddings in a PostgreSQL database with vector support using pgvector plugin, which can enable efficient information retrieval."],"when_not_to_use":["If your use case exclusively involves handling multimedia files like videos or images since ChatWeb is specifically designed for text-based content from web pages and documents.","Consider other tools if you are working in an environment where OpenAI services are not accessible or if APIs requiring personal API keys are a concern as this tool relies on the OpenAI API for its N-"],"source":"enrich:decision_facts","observed_at":"2026-07-11T03:39:59.118Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"ChatWeb is a Python-based content crawler and summarization chatbot designed for extracting main contents from web pages, PDFs, DOCX, and TXT files. It leverages OpenAI's GPT technology to summarize key points or answer질"}]}}