Results for “crawler”

markdown-crawler

paulpierre/markdown-crawler

A multithreaded web crawler that converts websites to markdown files for LLM RAG

Scrapegraph-ai

ScrapeGraphAI/Scrapegraph-ai

A Python scraping library with LLM and direct graph logic

firecrawl

firecrawl/firecrawl

The API to search, scrape, and interact with the web at scale.

trafilatura

adbar/trafilatura

Python & Command-line tool to gather web text and metadata

chatWeb

SkywalkerDarren/chatWeb

Web crawling, text extraction from PDF/DOCX/TXT files, summarization, and Q&A based on GPT3.5 embedding API

autoflow

pingcap/autoflow

Graph RAG based and conversational knowledge base tool