Generative AI feels like magic, but underneath it's linear algebra, calculus, probability theory, and statistics. Explore each layer of how models like GPT, Claude, and Gemini actually work — from raw text to generated output.
Every step above is powered by mathematics. Click “Start Exploring” to see each one in detail.
Turning words into numbers
How it works: Before any math happens, text must be converted to numbers. Tokenizers split text into subword pieces using algorithms like Byte-Pair Encoding (BPE). Each token maps to an integer ID — this is the model's actual input. A typical LLM vocabulary has 50,000-100,000 tokens.
Words as vectors in space
How it works: Each token is mapped to a high-dimensional vector (e.g., 768 or 4096 dimensions). Words with similar meanings cluster together. These vectors are learned during training — the model adjusts them so that the geometric relationships between vectors capture semantic relationships between words. This is pure linear algebra.
How tokens relate to each other
| Q \ K | The | cat | sat | on | the | mat |
|---|---|---|---|---|---|---|
| The | 80 | 5 | 5 | 3 | 4 | 3 |
| cat | 10 | 50 | 15 | 5 | 5 | 15 |
| sat | 5 | 35 | 40 | 10 | 2 | 8 |
| on | 3 | 10 | 15 | 50 | 12 | 10 |
| the | 30 | 2 | 3 | 15 | 40 | 10 |
| mat | 2 | 5 | 8 | 15 | 10 | 60 |
How it works: Self-attention lets each token “look at” every other token and decide how much to focus on each one. It's computed via matrix multiplication and softmax (a probability function). This is the core innovation of the Transformer architecture that powers GPT, Claude, and all modern LLMs.
The computational backbone
How it works: A neural network is layers of matrix multiplications followed by non-linear activation functions. Each “parameter” is just a number in a weight matrix. GPT-4 has ~1.8 trillion of these numbers — all learned through training. The “intelligence” is entirely encoded as matrices of floating-point numbers.
Generating text one token at a time
How it works: The model outputs a score (logit) for every possible next token. The softmax function converts these to a probability distribution. Temperature controls randomness — lower values make the model more deterministic, higher values more creative. The next token is then sampled from this distribution. It's all probability theory.
How models learn from data
Feed data through the network (matrix multiplications)
Measure how wrong the prediction is (cross-entropy loss)
Compute gradients using the chain rule (calculus!)
Adjust parameters to reduce loss (gradient descent)
How it works: Training is an optimization loop powered by calculus. The model makes a prediction, measures the error (loss), then uses backpropagation (chain rule from calculus) to figure out how to adjust each of its billions of parameters. This repeats millions of times on massive datasets. Training GPT-4 reportedly cost >$100M in compute.
Every “intelligent” response from an AI model is the result of billions of floating-point numbers being multiplied, added, and transformed through mathematical functions.
Linear algebra gives us the computational framework. Calculus lets us optimize. Probability & statistics make predictions possible. Information theory tells us what to optimize for.
AI didn't replace math — math created AI.
Matrix multiplications power every layer. Embeddings, attention, and weights are all matrices and vectors.
Backpropagation uses the chain rule to compute gradients. Without calculus, models can't learn.
Softmax, sampling, cross-entropy loss — the model outputs and learns through probability distributions.
Gradient descent, Adam optimizer, learning rate schedules — finding the minimum of the loss landscape.
“Mathematics is the queen of the sciences.”
— Carl Friedrich Gauss
The greatest thing is not AI — it's the mathematics that makes AI possible. Every breakthrough in AI is fundamentally a breakthrough in applied mathematics.