“corpus”

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

gpl

Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled corpus and yields massive improvement: "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation o

awesome-persian-nlp-ir

Curated List of Persian Natural Language Processing and Information Retrieval Tools and Resources

chunktuner

Benchmark and optimize chunking strategies for RAG corpus