Building GPT from Scratch: Part 1
Published: Jul 22, 2026 | Author: Kshayik Doshi
This is the first article in a series where I attempt to build a GPT-style language model from scratch. I'll first understand the mathematics behind it, then rebuild it in Modern C++, optimize it for the CPU, and finally write CUDA kernels to run it on the GPU.
Prerequisites: A little bit of knowledge on how computers work, a general idea about AI/ML models, and a whole lot of curiosity.
Background
Every engineer eventually develops a preference for the kinds of problems they enjoy solving, even though engineers are generally good at solving anything you throw at them. Over the past couple of years, the projects that I've built belonged to two different worlds:
- Systems Engineering: I love writing C++ code, interacting directly with hardware, and having deep control over the machine. Squeezing every last bit of performance out of the hardware and trying to run stuff that isn't supposed to run on old hardware is what drives me.
- AI Engineering: My projects usually revolve around intelligent systems, LLMs, agentic pipelines, and making AI products. But I was always curious: what’s actually inside the LLM that I'm building around?
Bring these two worlds together and boom — here enters this project and this series!
Diving Deep
Before starting, I did have a decent high-level idea of how LLMs worked: the input gets broken into small, sub-word level tokens (something computers can understand numerically), and then the model predicts probabilities for what token should appear after the current one. Repeat this 100 times, and you get a couple of lines.
I wasn't wrong, but this was surface-level knowledge. As someone who loves to dive deep into systems, I thought: it’s time to actually get into the thick of it.
For what felt like forever, Andrej Karpathy's "Let's Build GPT: From Scratch" lecture had been sitting in my YouTube Watch Later playlist. Eventually, "someday" became now. Alongside the lecture, I also started reading the original "Attention Is All You Need" paper.
The Core Objective
A language model has a deceptively simple objective: "Given everything that you've seen so far, predict the next token." That’s literally it.
1. Encoding & Vocabulary
Computers don't understand English words. Input text is converted into numbers. Something like 'a' becomes 0, 'b' becomes 1, and so on. This mapping depends entirely on the vocabulary size.
2. Chunking: The Model Never Reads Shakespeare at Once
The text is split into thousands of tiny chunks of fixed maximum length called the context window or block_size. Suppose chunk size is 4. We pull chunk length 4 + 1 = 5 tokens. Input: Tokens 0 to block_size - 1. Target: Tokens 1 to block_size.
3. Tensors: [B x T x C]
Three dimensions describe data shape flowing through the model: B (Batch Size), T (Time / Sequence Length), C (Channels / Embedding Dimension). The 3D tensor [B x T x C] travels through the entire transformer architecture.
4. Loss & Text Generation
Loss function score (Cross Entropy Loss) measures prediction performance. Autoregressive generation takes tokens, predicts probability distribution for next token, takes prediction for last token, appends it, and repeats recursively.
5. The Current Bottleneck
Baseline bigram models only look up local character frequencies. Recurrent architectures (RNNs, LSTMs) tried passing hidden states sequentially, but suffered from sequential bottlenecks (cannot parallelize on GPUs) and vanishing context. Self-attention from "Attention Is All You Need" solved both.
Transformers and Self Attention (Building GPT from Scratch: Part 2)
Published: Jul 23, 2026 | Author: Kshayik Doshi
Hi, I’m Kshayik Doshi, pursuing Computer Engineering at DJ Sanghvi in my final year. I spend my time building low-level systems in C++ and hacking together AI products and agentic pipelines.
Why We Needed GPT & Self-Attention
Historically, researchers tried solving context using RNNs and LSTMs passing a hidden state sequentially, hitting two walls: sequential bottleneck (cannot parallelize) and vanishing context (telephone game). We needed every token in context to talk to every other token at the exact same time, fully parallelized. Enter Self-Attention.
Static vs. Contextual Embeddings
Embeddings group similar concepts close together in multi-dimensional space. Static embeddings assign the exact same vector for 'Apple' whether it means Apple phone (tech company) or Apple fruit. Self-Attention converts static embeddings into contextual embeddings by allowing tokens to look at surrounding words and dynamically update their representations.
Dissecting Self-Attention & The QKV Engine
Self-Attention enables models to understand context by relating different words within a sequence to one another. Defined by: Attention(Q, K, V) = softmax( (Q * K^T) / sqrt(d_k) ) * V.
- Query (Q): "What am I looking for?"
- Key (K): "What information do I contain?"
- Value (V): "What content do I pass along if we match?"
Multiplying input tensor by learnable projection matrices W_Q, W_K, W_V produces Q, K, V vectors. Dividing dot products by sqrt(d_k) prevents softmax saturation and vanishing gradients.
Going Full GPU (Building GPT from Scratch: Part 4)
Published: Aug 17, 2026 | Author: Kshayik Doshi
We started by teaching a tiny model to predict the next token, then opened up the Transformer and understood what was actually happening inside attention. In Part 3, we built the C++ engine. Now we're going full GPU with CUDA kernels, parallel execution, quantization, and squeezing it all onto an ESP32.
Why We Need GPUs
CPUs are built for complex, low-latency, sequential branching tasks. GPUs are built for massive, high-throughput parallel tasks (like rendering pixels or multiplying matrices). Moving Transformer arithmetic onto parallel GPU threads is crucial to make training fast.
What is CUDA & PCIe Bottleneck
CUDA lets us write C++ directly for GPU hardware, executing kernels on thousands of threads simultaneously. Host (CPU) and Device (GPU) don't share memory. Copying data across the PCIe bus is slow, so the golden rule is: send the data once, keep everything inside VRAM for the entire forward pass, and copy the predicted tokens back at the end.
Quantization & ESP32 Autoregressive inference
Quantization compresses floating-point weights into fewer bits (like 8-bit INT8 or 4-bit INT4) using a scale factor, losing tiny precision but dropping the memory footprint massively. This allows us to pack a 4.66 MB quantized model inside an ESP32 microcontroller with 520 KB of internal SRAM, using pre-allocated activations buffer pools.