{"data":{"slug":"jaymody-picogpt","name":"picoGPT","tagline":"An unnecessarily tiny implementation of GPT-2 in NumPy","github_url":"https://github.com/jaymody/picoGPT","owner":"jaymody","repo":"picoGPT","owner_avatar_url":"https://avatars.githubusercontent.com/u/26451316?v=4","primary_language":"Python","stars":3470,"forks":456,"topics":["deep-learning","gpt","gpt-2","large-language-models","machine-learning","neural-network","nlp","python"],"archived":false,"github_pushed_at":"2023-04-24T20:05:53+00:00","maintenance_label":"Dormant","stars_delta_30d":3,"url":"https://www.graphcanon.com/tools/jaymody-picogpt","markdown_url":"https://www.graphcanon.com/tools/jaymody-picogpt.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jaymody-picogpt","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jaymody-picogpt","description":"An unnecessarily tiny implementation of GPT-2 in NumPy.","homepage_url":null,"license":"MIT","open_issues":14,"watchers":26,"ai_summary":"PicoGPT is a minimal and readable GPT-2 model implementation using NumPy. The forward pass code is extremely concise (40 lines), but lacks efficiency features like batch processing or advanced sampling techniques.","readme_excerpt":"# PicoGPT\nAccompanying blog post: [GPT in 60 Lines of Numpy](https://jaykmody.com/blog/gpt-from-scratch/)\n\n---\n\nYou've seen [openai/gpt-2](https://github.com/openai/gpt-2).\n\nYou've seen [karpathy/minGPT](https://github.com/karpathy/mingpt).\n\nYou've even seen [karpathy/nanoGPT](https://github.com/karpathy/nanogpt)!\n\nBut have you seen [picoGPT](https://github.com/jaymody/picoGPT)??!?\n\n`picoGPT` is an unnecessarily tiny and minimal implementation of [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) in plain [NumPy](https://numpy.org). The entire forward pass code is [40 lines of code](https://github.com/jaymody/picoGPT/blob/main/gpt2_pico.py#L3-L41).\n\npicoGPT features:\n* Fast? ❌ Nah, picoGPT is megaSLOW 🐌\n* Training code? ❌ Error, 4️⃣0️⃣4️⃣ not found\n* Batch inference? ❌ picoGPT is civilized, single file line, one at a time only\n* top-p sampling? ❌ top-k? ❌ temperature? ❌ categorical sampling?! ❌ greedy? ✅\n* Readable? `gpt2.py` ✅ `gpt2_pico.py` ❌\n* Smol??? ✅✅✅✅✅✅ YESS!!! TEENIE TINY in fact 🤏\n\nA quick breakdown of each of the files:\n\n* `encoder.py` contains the code for OpenAI's BPE Tokenizer, taken straight from their [gpt-2 repo](https://github.com/openai/gpt-2/blob/master/src/encoder.py).\n* `utils.py` contains the code to download and load the GPT-2 model weights, tokenizer, and hyper-parameters.\n* `gpt2.py` contains the actual GPT model and generation code which we can run as a python script.\n* `gpt2_pico.py` is the same as `gpt2.py`, but in even fewer lines of code. Why? Because why not 😎👍.\n\n#### Dependencies\n```bash\npip install -r requirements.txt\n```\nTested on `Python 3.9.10`.\n\n#### Usage\n```bash\npython gpt2.py \"Alan Turing theorized that computers would one day become\"\n```\n\nWhich generates\n\n```\n the most powerful machines on the planet.\n\nThe computer is a machine that can perform complex calculations, and it can perform these calculations in a way that is very similar to the human brain.\n```\n\nYou can also control the number of tokens to generate, the model size (one of `[\"124M\", \"355M\", \"774M\", \"1558M\"]`), and the directory to save the models:\n\n```bash\npython gpt2.py \\\n    \"Alan Turing theorized that computers would one day become\" \\\n    --n_tokens_to_generate 40 \\\n    --model_size \"124M\" \\\n    --models_dir \"models\"\n```","github_created_at":"2023-01-21T21:07:13+00:00","created_at":"2026-07-07T17:35:44.114428+00:00","updated_at":"2026-08-18T00:01:53.332069+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpt","name":"gpt"},{"slug":"gpt-2","name":"gpt-2"},{"slug":"large-language-models","name":"large language models"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"neural-network","name":"neural-network"},{"slug":"nlp","name":"nlp"}],"trust":{"provenance":{"is_fork":false,"github_id":591782276,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-18T00:01:52.592Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1211,"last_release_at":null,"stars_delta_30d":3,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":32,"high_count":0,"last_scan_at":"2026-07-11T11:06:02.571Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-18T00:01:53.052Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-18T00:01:53.052Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-18T00:01:53.052Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["PicoGPT may struggle with larger datasets due to its inefficiencies, despite being minimal."],"min_ram_gb":2,"requires_docker":false},"constraints":{"min_ram_gb":2,"requires_docker":false},"when_to_use":["- Use `picoGPT` when you need an example to understand GPT-2's functioning at its most pared-down level.","- Opt for this tool if your goal is educational or as a base to elaborate on for academic demonstration without focusing on performance."],"when_not_to_use":["- Avoid `picoGPT` in scenarios requiring efficient batch processing or advanced generation techniques like top-p sampling, as it lacks these features.","- Do not use `picoGPT` if speed and scalability are critical for your project, given its megaSlow execution."],"source":"enrich:decision_facts","observed_at":"2026-07-12T14:05:50.457Z"},"constraint_facets":{"min_ram_gb":2,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Min 2 GB RAM; PicoGPT may struggle with larger datasets due to its inefficiencies, despite being minimal."},{"label":"Adopt for","value":"`picoGPT` is a minimal and extremely compact GPT-2 model, written in NumPy for the sake of readability despite significant inefficiencies."},{"label":"License detail","value":"`MIT License` - A permissive license enabling free modification and distribution even in commercial software."}]}}