Build Your Own LLM — From Scratch

Try It: Bigram Prediction

You just did language modeling! You used your knowledge to predict the next word. A **bigram model** does the same thing with statistics.

Building a Bigram Table

**The AI picks the most likely next word based on what it saw in training.** But there's a problem...

// Bigram problem: only looks at ONE previous word
"The cat sat on the ___"
// Only sees "the", ignores "cat sat on"
// Might predict "cat" again!

Bigrams are dumb. They only look at one word back. Neural networks can look at ALL previous words at once.

Neural Network Magic

Instead of counting words, a **neural network learns weights** — numbers that capture patterns in language.

// Simplified neural network prediction
function predict(tokens, weights) {
  let score = 0;
  
  // Each token contributes to the prediction
  for (let i = 0; i < tokens.length; i++) {
    let tokenId = tokens[i];
    let position = i;
    
    // Learned weights for this token at this position
    score += weights[tokenId][position];
  }
  
  // Convert score to probability
  return softmax(score);
}

// The network learns: what patterns predict what words?
// "The cat sat" → high score for "on"
// "I love" → high score for "you"

Training adjusts the weights until predictions match real text. See 1 billion examples, learn 1 billion patterns.

Try It: Attention Heatmap

Notice how "sat" pays attention to "cat" (the thing doing the sitting) and "on" pays attention to "mat" (the location)? The network learns these relationships automatically.

The Math (With Real Numbers)

**Attention uses three matrices: Query (Q), Key (K), and Value (V).** Think of it like a database lookup:

// Example with tiny numbers:
// Word: "cat"
let query = [0.1, 0.8];    // "What am I looking for?"
let key   = [0.2, 0.9];    // "What do I contain?"  
let value = [0.5, 0.3];    // "What information do I have?"

// Attention score = query · key (dot product)
let score = query[0]*key[0] + query[1]*key[1]
         = 0.1*0.2 + 0.8*0.9 
         = 0.02 + 0.72 
         = 0.74

// High score = pay attention to this word

**The scaled dot-product attention equation:**

function attention(Q, K, V) {
  // 1. Calculate all attention scores
  let scores = matmul(Q, transpose(K));
  
  // 2. Scale by sqrt(dimension) to prevent explosion
  scores = scale(scores, 1 / Math.sqrt(K[0].length));
  
  // 3. Softmax: turn scores into probabilities
  let weights = softmax(scores);
  
  // 4. Mix the values according to attention weights
  let output = matmul(weights, V);
  
  return output;
}

// This happens in parallel for every word in the sentence

Every word gets to vote on what every other word should represent. This is the breakthrough that made modern AI possible.

Try It: Layer by Layer

More layers = more complex thinking. But there's a pattern to how much better they get...

The Scaling Laws

Here's the most important discovery in AI: quality improves predictably with scale.

// The power law of language models:
Loss = A × (Compute)^(-α)

Where:
- Loss = how wrong the model is (lower = better)
- Compute = training budget (GPUs × time)  
- α ≈ 0.05 (the scaling exponent)

Translation: 10x more compute → predictably better AI

This is why everyone is building bigger models:

• GPT-1 (2018): 117M parameters, $600 training cost
• GPT-2 (2019): 1.5B parameters, $40K training cost
• GPT-3 (2020): 175B parameters, $4.6M training cost
• GPT-4 (2023): ~1.8T parameters, ~$100M training cost

The Chinchilla insight: Don't just make models bigger — train them on more data too. The optimal ratio is ~20 tokens per parameter.

GPT-4 is this exact architecture, just bigger. You now understand how it thinks.

Try It: Base vs Chat Model

Same model, different behavior! The chat version learned conversational format during training.

The Training Process

Making a model conversational requires three steps:

// Step 1: Instruction Tuning
// Show the model conversation examples:
[
  {
    "messages": [
      {"role": "user", "content": "What is Python?"},
      {"role": "assistant", "content": "Python is a programming language..."}
    ]
  },
  // ... thousands more examples
]

// Step 2: Human Feedback (RLHF)  
// Humans rank model responses:
// Response A: "Python is a programming language..." ⭐⭐⭐⭐⭐
// Response B: "Python is a snake that..." ⭐⭐

// Step 3: System Prompts
// Hidden instructions that shape behavior:
"You are a helpful, harmless, and honest assistant. 
Answer questions clearly and concisely."

The model learns: "When I see a conversation format, I should be helpful, not just continue the text."

You Built an LLM!

🎉 Congratulations! 🎉

You now understand every major component of modern language models:

• ✅ Tokenization: Converting text to numbers
• ✅ Prediction: Using patterns to guess next words
• ✅ Attention: Letting words talk to each other
• ✅ Scaling: More layers = more intelligence
• ✅ Chat Training: Teaching conversation format

GPT, Claude, Gemini — they're all variations of what you just learned. Different training data, different fine-tuning, but the same core architecture.

The next time someone asks "How does ChatGPT work?" — you can actually explain it.