---
title: "picoGPT"
type: "tool"
slug: "jaymody-picogpt"
canonical_url: "https://www.graphcanon.com/tools/jaymody-picogpt"
github_url: "https://github.com/jaymody/picoGPT"
homepage_url: null
stars: 3465
forks: 459
primary_language: "Python"
license: "MIT"
categories: ["llm-frameworks"]
tags: ["deep-learning", "nlp", "machine-learning", "neural-network", "python", "large-language-models", "gpt", "gpt-2"]
updated_at: "2026-07-07T20:01:44.425005+00:00"
---

# picoGPT

> An unnecessarily tiny implementation of GPT-2 in NumPy

picoGPT is a minimal and barebones implementation of the large-scale GPT-2 model, featuring just 40 lines for its forward pass, using only NumPy without high-level libraries. It includes tokenization, model loading utilities, and generation capabilities.

## Facts

- Repository: https://github.com/jaymody/picoGPT
- Stars: 3,465 · Forks: 459 · Open issues: 14 · Watchers: 26
- Primary language: Python
- License: MIT
- Last pushed: 2023-04-24T20:05:53+00:00

## Categories

- [LLM Frameworks](/categories/llm-frameworks.md)

## Tags

deep-learning, nlp, machine-learning, neural-network, python, large-language-models, gpt, gpt-2

## Related tools

- [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,664)
- [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,025)
- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,350)
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,582)
- [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702)
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,715)
- [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,619)
- [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,189)

## README (excerpt)

```text
# PicoGPT
Accompanying blog post: [GPT in 60 Lines of Numpy](https://jaykmody.com/blog/gpt-from-scratch/)

---

You've seen [openai/gpt-2](https://github.com/openai/gpt-2).

You've seen [karpathy/minGPT](https://github.com/karpathy/mingpt).

You've even seen [karpathy/nanoGPT](https://github.com/karpathy/nanogpt)!

But have you seen [picoGPT](https://github.com/jaymody/picoGPT)??!?

`picoGPT` is an unnecessarily tiny and minimal implementation of [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) in plain [NumPy](https://numpy.org). The entire forward pass code is [40 lines of code](https://github.com/jaymody/picoGPT/blob/main/gpt2_pico.py#L3-L41).

picoGPT features:
* Fast? ❌ Nah, picoGPT is megaSLOW 🐌
* Training code? ❌ Error, 4️⃣0️⃣4️⃣ not found
* Batch inference? ❌ picoGPT is civilized, single file line, one at a time only
* top-p sampling? ❌ top-k? ❌ temperature? ❌ categorical sampling?! ❌ greedy? ✅
* Readable? `gpt2.py` ✅ `gpt2_pico.py` ❌
* Smol??? ✅✅✅✅✅✅ YESS!!! TEENIE TINY in fact 🤏

A quick breakdown of each of the files:

* `encoder.py` contains the code for OpenAI's BPE Tokenizer, taken straight from their [gpt-2 repo](https://github.com/openai/gpt-2/blob/master/src/encoder.py).
* `utils.py` contains the code to download and load the GPT-2 model weights, tokenizer, and hyper-parameters.
* `gpt2.py` contains the actual GPT model and generation code which we can run as a python script.
* `gpt2_pico.py` is the same as `gpt2.py`, but in even fewer lines of code. Why? Because why not 😎👍.

#### Dependencies
```bash
pip install -r requirements.txt
```
Tested on `Python 3.9.10`.

#### Usage
```bash
python gpt2.py "Alan Turing theorized that computers would one day become"
```

Which generates

```
 the most powerful machines on the planet.

The computer is a machine that can perform complex calculations, and it can perform these calculations in a way that is very similar to the human brain.
```

You can also control the number of tokens to generate, the model size (one of `["124M", "355M", "774M", "1558M"]`), and the directory to save the models:

```bash
python gpt2.py \
    "Alan Turing theorized that computers would one day become" \
    --n_tokens_to_generate 40 \
    --model_size "124M" \
    --models_dir "models"
```
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/jaymody-picogpt`](/api/graphcanon/tools/jaymody-picogpt)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_