Building GPT from Scratch: Part 1

This is the first article in a series where I attempt to build a GPT-style language model from scratch. I'll first understand the mathematics behind it, then rebuild it in Modern C++, optimize it for the CPU, and finally write CUDA kernels to run it on the GPU.

Background

Every engineer eventually develops a preference for the kinds of problems they enjoy solving, even though engineers are generally good at solving anything you throw at them. Over the past couple of years, the projects that I've built belonged to two different worlds:

• Systems Engineering: I love writing C++ code, interacting directly with hardware, and having deep control over the machine. Squeezing every last bit of performance out of the hardware and trying to run stuff that isn't supposed to run on old hardware is what drives me.
• AI Engineering: My projects usually revolve around intelligent systems, LLMs, agentic pipelines, and making AI products. But I was always curious: what’s actually inside the LLM that I'm building around?

Bring these two worlds together and boom — here enters this project and this series!

Diving Deep

Before starting, I did have a decent high-level idea of how LLMs worked: the input gets broken into small, sub-word level tokens (something computers can understand numerically), and then the model predicts probabilities for what token should appear after the current one. Repeat this 100 times, and you get a couple of lines.

I wasn't wrong, but this was surface-level knowledge. As someone who loves to dive deep into systems, I thought: it’s time to actually get into the thick of it.

For what felt like forever, Andrej Karpathy's "Let's Build GPT: From Scratch" lecture had been sitting in my YouTube Watch Later playlist. Every few months I'd think, "I'll watch it someday." Eventually, "someday" became now.

Alongside the lecture, I also started reading the original "Attention Is All You Need" paper — because believe it or not, that's where this whole LLM mania started. Both are excellent resources if you're trying to get into it.

I grabbed my book and pen and started watching and reading again and again until I could finally understand what's going on below the hood: the math, the pipeline, everything. Here's what I noted down and understood.

(Disclaimer: I'll try to keep the ideas clear and as easy to understand as possible, but curiosity will help you more than I can!)

The Core Objective

A language model has a deceptively simple objective:

"Given everything that you've seen so far, predict the next token."

That’s literally it. This happens for every single token, and once a token is generated, it becomes part of the ongoing context for the next prediction.

Context is essential. For example, consider the sentence: "I bought an Apple..." Without context, "Apple" could mean a fruit or a tech company. Context gives words their exact meaning.

1. Encoding & Vocabulary

Computers don't understand English words, so how does a model know what an "apple" even is?

This is where encoding enters the picture. Input text is converted into numbers. Something like 'a' becomes 0, 'b' becomes 1, and so on. This mapping depends entirely on the vocabulary size (the total number of unique tokens the model knows).

For this initial study, I'll be considering character-level tokens, meaning a token will basically be a single character. This is as opposed to what current big models do (using sub-word level tokens, like splitting "retest" into "re" and "test"), which increases vocabulary size and implementation complexity but reduces the sequence length.

Here's how character encoding works:

We first decide the vocabulary size (the number of unique characters the model will ever see input/output-wise), and then based on that, the characters are mapped using an embedding lookup table (remember this, as it'll be useful throughout).

2. Chunking: The Model Never Reads Shakespeare at Once

One of the most surprising things I learned is that the model never receives the entire dataset at once. Initially, I imagined training looked something like this:

Reality is much more interesting. The text is split into thousands of tiny chunks. Each chunk becomes an independent training sample. These chunks have a fixed maximum length called the context window or block_size. The model only sees that much context while making its prediction.

One Chunk Teaches Multiple Lessons

Suppose the chunk size (block_size) is 4. We actually pull a chunk of length 4 + 1 = 5 tokens. Why?

We split this chunk into two sets:

• Input: Tokens from index 0 to block_size - 1
• Target: Tokens from index 1 to block_size

For a chunk [18, 47, 56, 57, 58]:

• Input: [18, 47, 56, 57]
• Target: [47, 56, 57, 58]

In execution, it looks like this:

The same single chunk teaches the model:

• How to predict with 1 token of context
• How to predict with 2 tokens of context
• How to predict with 3 tokens of context
• ...and so on.

Every chunk packs multiple learning opportunities into a single forward pass. It's an elegant idea that wasn't obvious to me before working through the implementation.

3. Tensors: [B x T x C]

Now, till this point, we've only talked about numbers. But neural networks don't just deal with one number at a time — they process huge collections of numbers together because that's what modern hardware (GPU and TPU) is ridiculously good at.

This is where matrices (or tensors) enter the picture. Throughout this project, you'll keep seeing three letters pop up everywhere:

These three dimensions pretty much describe the shape of the data flowing through the entire model:

• B (Batch Size): Instead of processing one sentence at a time, we process multiple chunks together. If my batch size is 32, I'm basically teaching the model using 32 different chunks simultaneously.
• T (Time / Sequence Length): This is simply how many tokens exist inside one chunk.
• C (Channels / Embedding Dimension): The most important of the three. Instead of token 17 being represented as just the number 17, it gets looked up inside an Embedding Lookup Table and becomes a dense vector like [0.23, -1.12, 0.84, ...].

This 3D tensor is basically what travels through almost the entire transformer architecture.

4. Loss & Text Generation

Now, obviously, we need some way of knowing whether the model is actually learning or if it's just confidently predicting absolute garbage.

This is where the loss function comes into the picture. A loss function is basically a score: the lower the score, the better the model is doing. I used a Negative Log Likelihood function, more commonly implemented as Cross Entropy Loss.

How Generation Works

Now that the model has learnt something, how does it actually generate text? Turns out, it's almost exactly the process we described at the beginning:

1. We take a chunk of tokens and perform a forward pass through the transformer.
2. The model predicts a probability distribution for the next token at every position in the sequence.
3. During generation, however, we don't care about all of those predictions — we only care about the prediction corresponding to the last token, because that's the one telling us what should come next.
4. Suppose the prompt is "Hello, my name is". The model internally predicts the next token for every position, but we only use the prediction made after "is", which might be "John". We append that token to the prompt, making the input "Hello, my name is John", and feed the entire sequence back into the model.

It predicts one more token, appends it again, and repeats the process until the response is complete. That's literally how GPT writes essays, code, and conversations: one token at a time, repeatedly extending its own context.

5. The Current Bottleneck

At this point, the model still isn't very smart though. In fact, it's surprisingly dumb. Like... what in the Shakespearean is this?

This happens because each prediction mainly depends on local lookup statistics it has learned.

Earlier architectures like RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory Networks) tried solving this problem by remembering information through something called a hidden state. Well, it did work, but the problem was easily visible: it was sequential since every token depended on the previous one.

And this is exactly where the famous paper "Attention Is All You Need" completely changed the game.

What's Next?

That’s already a ton to take in for one sitting, so I'm splitting this initial theory deep-dive into two parts!

Here’s the game plan for the rest of this series:

In the next part, we'll finally open the transformer itself and see why attention changed modern AI.