LLMDataHub: Awesome Datasets for LLM Training

---

🔥 Alignment Datasets • 💡 Domain-specific Datasets • :atom: Pretraining Datasets 🖼️ Multimodal Datasets

Introduction 📄

Large language models (LLMs), such as OpenAI's GPT series, Google's Bard, and Baidu's Wenxin Yiyan, are driving profound technological changes. Recently, with the emergence of open-source large model frameworks like LlaMa and ChatGLM, training an LLM is no longer the exclusive domain of resource-rich companies. Training LLMs by small organizations or individuals has become an important interest in the open-source community, with some notable works including Alpaca, Vicuna, and Luotuo. In addition to large model frameworks, large-scale and high-quality training corpora are also essential for training large language models. Currently, relevant open-source corpora in the community are still scattered. Therefore, the goal of this repository is to continuously collect high-quality training corpora for LLMs in the open-source community.

Training a chatbot LLM that can follow human instruction effectively requires access to high-quality datasets that cover a range of conversation domains and styles. In this repository, we provide a curated collection of datasets specifically designed for chatbot training, including links, size, language, usage, and a brief description of each dataset. Our goal is to make it easier for researchers and practitioners to identify and select the most relevant and useful datasets for their chatbot LLM training needs. Whether you're working on improving chatbot dialogue quality, response generation, or language understanding, this repository has something for you.

Contact 📬

If you want to contribute, you can contact:

Junhao Zhao 📧
Advised by Prof. Wanyun Cui

General Open Access Datasets for Alignment 🟢:

Type Tags 🏷️:

• SFT: Supervised Finetune
  • Dialog: Each entry contains continuous conversations
  • Pairs: Each entry is an input-output pair
  • Context: Each entry has a context text and related QA pairs
• PT: pretrain
• CoT: Chain-of-Thought Finetune
• RLHF: train reward model in Reinforcement Learning with Human Feedback

Datasets released in November 2023

Dataset name	Used by	Type	Language	Size	Description ️
helpSteer	/	RLHF	English	37k instances	An RLHF dataset that is annotated by human with helpfulness, correctness, coherence, complexity and verbosity measures
no_robots	/	SFT	English	10k instance	High-quality human-created STF data, single turn.

Datasets released in September 2023

| Dataset name