{"data":{"slug":"fareedkhan-dev-train-llm-from-scratch","name":"train-llm-from-scratch","tagline":"A straightforward method for training your LLM from raw text to aligned model generation","github_url":"https://github.com/FareedKhan-dev/train-llm-from-scratch","owner":"FareedKhan-dev","repo":"train-llm-from-scratch","owner_avatar_url":"https://avatars.githubusercontent.com/u/63067900?v=4","primary_language":"Python","stars":9141,"forks":1264,"topics":["gemini","large-language-models","llm","openai","training","transformers"],"archived":false,"github_pushed_at":"2026-08-17T05:07:26+00:00","maintenance_label":"Very active","stars_delta_30d":765,"url":"https://www.graphcanon.com/tools/fareedkhan-dev-train-llm-from-scratch","markdown_url":"https://www.graphcanon.com/tools/fareedkhan-dev-train-llm-from-scratch.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/fareedkhan-dev-train-llm-from-scratch","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=fareedkhan-dev-train-llm-from-scratch","description":"A straightforward method for training your LLM, from downloading data to generating text.","homepage_url":"https://fareedkhan-dev.github.io/train-llm-from-scratch/","license":"MIT","open_issues":6,"watchers":69,"ai_summary":"Provides scripts to train large language models (LLM) with various parameters using PyTorch, starting from basic data processing and ending at an aligned reasoning style model.","readme_excerpt":"<div align=\"center\">\n\n\n# Train LLM From Scratch\n\n   \n\n**I am Looking for a PhD position in AI**. [GitHub](https://github.com/FareedKhan-dev)\n\n</div>\n\nI implemented a transformer model from scratch using PyTorch, based on the paper [Attention is All You Need](https://arxiv.org/abs/1706.03762). You can use my scripts to train your own **billion** or **million** parameter LLM using a single GPU.\n\nThis started as a pretraining tutorial. It now goes all the way from raw text to an aligned, reasoning style model, with every algorithm hand written in plain PyTorch (no `trl`, no `peft`, no `transformers`). The whole journey is one idea repeated: turn text into numbers, predict the next token, then keep changing the data and the loss until the model does what we want.\n\n\n\nHere is the path we will walk, end to end:\n\n```\nraw text  ->  tokens  ->  a Transformer  ->  next-token loss  ->  a base model\nbase model  ->  SFT  ->  Reward Model  ->  {PPO, DPO}  ->  GRPO  ->  evaluation and chat\n```\n\nBelow is the output of a trained 13 million parameter LLM, just so you can see where the small end of this starts:\n\n```\nIn ***1978, The park was returned to the factory-plate that\nthe public share to the lower of the electronic fence that\nfollow from the Station's cities. The Canal of ancient Western\nnations were confined to the city spot. The villages were directly\nlinked to cities in China that revolt that the US budget and in\nOdambinais is uncertain and fortune established in rural areas.\n```\n\n\n## Table of Contents\n- [Who this is for](#who-this-is-for)\n- [Prerequisites and Training Time](#prerequisites-and-training-time)\n- [Setup](#setup)\n- [Code Structure](#code-structure)\n- [Step 1: Preparing the Data](#step-1-preparing-the-data)\n- [Step 2: The Model, Built From Small Pieces](#step-2-the-model-built-from-small-pieces)\n  - [Multi Layer Perceptron (MLP)](#multi-layer-perceptron-mlp)\n  - [Single Head Attention](#single-head-attention)\n  - [Multi Head Attention](#multi-head-attention)\n  - [The Transformer Block](#the-transformer-block)\n  - [The Full Transformer](#the-full-transformer)\n- [Step 3: Pretraining the Base Model](#step-3-pretraining-the-base-model)\n- [Step 4: Generating Text](#step-4-generating-text)\n- [Step 5: Post-Training, Turning a Base Model Into an Assistant](#step-5-post-training-turning-a-base-model-into-an-assistant)\n  - [SFT (Supervised Fine-Tuning)](#sft-supervised-fine-tuning)\n  - [The Reward Model](#the-reward-model)\n  - [DPO, ORPO and KTO](#dpo-orpo-and-kto)\n  - [PPO](#ppo)\n  - [GRPO / RLVR](#grpo--rlvr)\n- [Step 6: Evaluation](#step-6-evaluation)\n- [Step 7: Talking to the Model](#step-7-talking-to-the-model)\n- [The Streamlit Control Panel](#the-streamlit-control-panel)\n- [The Documentation Site](#the-documentation-site)\n- [Run the Whole Thing](#run-the-whole-thing)\n- [What's Next](#whats-next)\n\n## Who this is for\n\nI tried to write this so one page works for very different readers:\n\n- If you are a **student**, read top to bottom. Every block of code comes after a plain explanation of what it does and why, and most blocks are followed by the output you should expect.\n- If you are a **developer**, the commands and file paths are all here. You can copy, run, and read the referenced source files directly.\n- If you are a **researcher**, the post-training half is the interesting part: SFT, a Bradley-Terry reward model, PPO with GAE, DPO/ORPO/KTO, and GRPO, all from scratch on the same small Transformer, trained on real public datasets.\n\nEvery diagram in this README is colored the same way, so the colors mean something:\n\n- green is raw data\n- teal is stored, tokenized data on disk\n- blue is a plain processing step\n- yellow is the model or a training step\n- orange is the reinforcement learning and reward parts\n- red is a loss\n- grey is a saved checkpoint\n- purple is the final output or evaluation\n\n## Prerequisites and Training Time\n\nYou need a basic understanding of object oriented programming, neural networks, and PyTorch. Below are so","github_created_at":"2025-01-12T04:03:05+00:00","created_at":"2026-07-07T17:34:22.943581+00:00","updated_at":"2026-08-17T06:02:14.716666+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"gemini","name":"gemini"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm","name":"llm"},{"slug":"openai","name":"openai"},{"slug":"transformers","name":"transformers"}],"trust":{"provenance":{"is_fork":false,"github_id":915520679,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-17T06:02:13.941Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":0,"last_release_at":null,"stars_delta_30d":765,"open_issues_delta_30d":4},"security_summary":{"status":"ok","scanner":"osv@v1","low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:03:00.155Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-17T06:02:14.435Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-17T06:02:14.435Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-17T06:02:14.435Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"This repository is available under the MIT license, allowing free use for both personal and commercial purposes. The model training requires resources on your end with no additional licensing costs."},"requirements":{"notes":["A single GPU environment is necessary.","Basic understanding of PyTorch is recommended to leverage the full potential of this tool.","Familiarity with NLP and transformer-based models can be helpful but not mandatory."],"min_ram_gb":null},"constraints":{"min_ram_gb":null,"pricing_model":"freemium"},"when_to_use":["You're interested in building an LLM from the ground up without relying on prebuilt packages like transformers or peft.","You are a student who wants to understand every step of creating and training an LLM through clear explanations and hand-written code.","You need to train a model using only PyTorch, avoiding dependencies such as trl or peft.","You're looking for a detailed guide that takes you from raw text data all the way to a functional aligned reasoning model."],"when_not_to_use":["Your goal is to rapidly prototype and fine-tune an existing pre-trained LLM with minimal coding effort.","You prefer using established transformer libraries or frameworks like Hugging Face's transformers, which offer quicker setup but less control over the underlying code.","You are working in a multi-GPU environment and need distributed training capabilities that go beyond what is offered here.","You seek immediate access to state-of-the-art models without wanting to dive into the intricate workings of an LLM."],"source":"enrich:decision_facts","observed_at":"2026-07-09T08:03:52.845Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"freemium"},"decision_summary":[{"label":"Pricing","value":"freemium - This repository is available under the MIT license, allowing free use for both personal and commercial purposes. The model training requires resources on your end with no additional licensing costs."},{"label":"Requirements","value":"A single GPU environment is necessary.; Basic understanding of PyTorch is recommended to leverage the full potential of this tool.; Familiarity with NLP and transformer-based models can be helpful but not mandatory."},{"label":"Adopt for","value":"train-llm-from-scratch offers a comprehensive approach for training your own Large Language Model (LLM) using PyTorch, solely powered by a single GPU."}]}}