{"data":{"slug":"paulpierre-markdown-crawler","name":"markdown-crawler","tagline":"A multithreaded web crawler for creating markdown files from website pages","github_url":"https://github.com/paulpierre/markdown-crawler","owner":"paulpierre","repo":"markdown-crawler","owner_avatar_url":"https://avatars.githubusercontent.com/u/142327?v=4","primary_language":"Python","stars":467,"forks":54,"topics":["html-to-markdown","html-to-markdown-converter","html2md","llm","llmops","markdown","markdown-crawler","markdown-parser","markdown-scraper","md-crawler","rag","web-scraper"],"archived":false,"github_pushed_at":"2026-06-26T14:23:38+00:00","maintenance_label":"Steady","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/paulpierre-markdown-crawler","markdown_url":"https://www.graphcanon.com/tools/paulpierre-markdown-crawler.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/paulpierre-markdown-crawler","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=paulpierre-markdown-crawler","description":"A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG","homepage_url":"https://pypi.org/project/markdown-crawler/","license":"MIT","open_issues":5,"watchers":5,"ai_summary":"PaulPierre/markdown-crawler is a Python-based tool that crawls websites and converts each page into a markdown file, aimed at facilitating the use of these documents in LLM RAG systems.","readme_excerpt":"# ⚠️  Requirements\n- Python 3.x\n- BeautifulSoup4\n- requests\n- markdownify\n\n<br><br>\n\n---\n\n# ⚖️ License\nMIT License\nCopyright (c) 2023 Paul Pierre\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\n<br><br>","github_created_at":"2023-10-24T13:30:06+00:00","created_at":"2026-07-07T17:43:35.593104+00:00","updated_at":"2026-08-21T06:01:51.298817+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"}],"tags":[{"slug":"html-to-markdown","name":"html-to-markdown"},{"slug":"html2md","name":"html2md"},{"slug":"llmops","name":"llmops"},{"slug":"markdown-crawler","name":"markdown-crawler"},{"slug":"web-scraper","name":"web-scraper"}],"trust":{"provenance":{"is_fork":false,"github_id":709317996,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-21T06:01:50.471Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":1,"days_since_push":55,"last_release_at":"2026-06-26T14:23:38Z","stars_delta_30d":2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:24:03.137Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-21T06:01:50.965Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-21T06:01:50.965Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-21T06:01:50.965Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-21T06:01:50.965Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["This tool requires Python 3.x in addition to specific libraries like BeautifulSoup4, requests, and markdownify."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["When you need to prepare website contents as structured markdown documents for usage in Retrieval-Augmented Generation (RAG) setups involving Language Learning Models (LLMs).","For projects where multithreaded crawling is desired to speed up the process of converting large volumes of web pages into a more machine-readable format like Markdown.","If you're working with Python and your project requires flexibility in customization due to an open-source MIT license, allowing for modifications without restrictions."],"when_not_to_use":["When the need arises for specific formats other than markdown, as this crawler is uniquely tailored towards generating content in that format.","For environments where multithreading could lead to issues or conflicts with the architecture of your existing system; if you experience problems with resource management under heavy load due to this."],"source":"enrich:decision_facts","observed_at":"2026-07-11T03:52:31.772Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"This tool requires Python 3.x in addition to specific libraries like BeautifulSoup4, requests, and markdownify."},{"label":"Adopt for","value":"markdown-crawler is a multithreaded Python-based web crawler aimed at converting web content into markdown format for integration with LLM RAG systems."},{"label":"License detail","value":"The software is provided under the MIT License, permitting free use and modification as long as the copyright notice and permission notice are included. There are no warranties offered for the license"}]}}