{"data":{"slug":"keyvank-femtogpt","name":"femtoGPT","tagline":"Pure Rust implementation of a minimal Generative Pretrained Transformer","github_url":"https://github.com/keyvank/femtoGPT","owner":"keyvank","repo":"femtoGPT","owner_avatar_url":"https://avatars.githubusercontent.com/u/4275654?v=4","primary_language":"Rust","stars":934,"forks":66,"topics":["from-scratch","gpt","gpu","hacktoberfest","llm","machine-learning","neural-network","opencl","rust"],"archived":false,"github_pushed_at":"2025-10-21T11:13:42+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/keyvank-femtogpt","markdown_url":"https://www.graphcanon.com/tools/keyvank-femtogpt.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/keyvank-femtogpt","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=keyvank-femtogpt","description":"Pure Rust implementation of a minimal Generative Pretrained Transformer","homepage_url":"https://discord.gg/wTJFaDVn45","license":"MIT","open_issues":10,"watchers":11,"ai_summary":null,"readme_excerpt":"# :robot: femtoGPT\n\n\n\n\n\nfemtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer.\n\nIt can be used for both *inference* and *training* of GPT-style language-models\nusing **CPUs** and **GPUs**!\n\n(***HEY!*** I'm also writing a book, which will soon discuss the implementation of a LLM in detail! Check it out here: [The Super Programmer](https://github.com/keyvank/tsp))\n\n## Usage\n\nTraining:\n\n`cargo run --release -- train`\n\nInference:\n\n`cargo run --release -- infer`\n\n(Note: Add `--features gpu` in order to leverage GPU speedups!)\n\n## Intro\n\nEverything is implemented from scratch, including the tensor processing logic\nalong with training/inference code of a minimal GPT architecture.\n\nThe architecture is very similar/almost identical with Andrej Karpathy's\n[nanoGPT video lecture](https://github.com/karpathy/ng-video-lecture).\n\nfemtoGPT is a great start for those who are fascinated by LLMs and would like to\nunderstand how these models work in very deep levels.\n\nfemtoGPT uses nothing but random generation libraries (`rand`/`rand-distr`), data-serialization\nlibraries (`serde`/`bincode` for saving/loading already trained models) and a\nparallel computing library (`rayon`).\n\nfemtoGPT is ~~EXTREMELY SLOW~~ ***relatively fast on CPU 😉***, and most of the\nprimitive operations (E.g Matrix multiplication) are implemented in the simplest way possible.\n\nCorrectness of gradients is checked using gradient-check method, though it still is very\npossible that some layers are implemented wrongly.\n\n([Discord server](https://discord.gg/wTJFaDVn45) for discussions around the project!)\n\n## Usage\n\nMake sure you have the Rust toolchain on your system, in order to compile and run\nthe project:\n\n`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`\n\nIf you want to train using a GPU, you will first need to make sure your GPU drivers\nare correctly installed on your system, and their OpenCL runtimes are available.\n\nOn Debian systems, you can setup OpenCL runtimes by installing the package `ocl-icd-opencl-dev`:\n\n`sudo apt install ocl-icd-opencl-dev`\n\n***GOOD NEWS!*** *Since femtoGPT's GPU implementation is based on OpenCL, it can\nrun on both NVIDIA and AMD cards, and you won't need to install heavy-weight\nCUDA-toolkits on your system. OpenCL runtimes would suffice!*\n\nNow you'll just need to put the text you want to train your GPT model on, inside\n`dataset.txt`. Make sure it has a small number of unique characters! (E.g. the\ncurrent dataset has only used 65 different unique characters!)\n\nThen you'll need to run:\n\n```\ncargo run --release\n```\n\nIt will start training the model and will put the training data in the `train_data`\ndirectory. You can stop the training and continue later!\n\n## Output samples\n\nAfter hours of training on the Shakespeare database, on a 300k parameter model,\nthis has been the output:\n\n```\nLIS:\nTore hend shater sorerds tougeng an herdofed seng he borind,\nOund ourere sthe, a sou so tousthe ashtherd, m se a man stousshan here hat mend serthe fo witownderstesther s ars at atheno sel theas,\nthisth t are sorind bour win soutinds mater horengher\n```\n\nThis is embarrassingly bad, but looking at the bright side, it seems like it has\nbeen able to generate words that are easy to pronounce.\n\nI'm currently training a 10M parameter model to further examine the correctness\nof my implementation.\n\n**UPDATE 5th June 2023:**\n\nThis has been a new output, after more hours of training on a model with similar scale:\n\n```\nWhat like but wore pad wo me che nogns yous dares,\nAs supt it nind bupart 'the reed:\nAnd hils not es\n```\n\nObviously the model has started to learn some words and punctuation rules!\n\n**UPDATE 9th June 2023:**\n\nModel was able to reach loss value of ~1.4\n\nHere is an example output:\n\n```\nAdistition gone; true; schistoes for mine souls!\nBefore your home, bariechts should be\nCarlam on that's a worf quirer of him so.\nWhat look'd lack away more\nTo him foot; one hour fortious of saves:\nSon;\n'Tis all Earl mmistling me.\n\nHAR","github_created_at":"2023-05-28T17:41:48+00:00","created_at":"2026-07-11T10:42:12.027678+00:00","updated_at":"2026-07-11T10:42:19.907673+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"gpu","name":"gpu"},{"slug":"llm","name":"llm"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"neural-network","name":"neural-network"},{"slug":"hacktoberfest","name":"hacktoberfest"},{"slug":"opencl","name":"opencl"},{"slug":"from-scratch","name":"from-scratch"},{"slug":"gpt","name":"gpt"}],"trust":{"provenance":{"is_fork":false,"github_id":646532460,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:42:12.629Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":262,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:42:13.967Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:42:13.727Z"},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-07-11T10:42:13.727Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T10:42:13.727Z"}}}}