5 Pieces of Math Every AI Actually Runs On
@_yusufknl · @xx_video_download_bot
I'm @_yusufknl - AI researcher by day, and I break down the math the hype skips. If this is your kind of thing, the follow is the move: I post this stuff weekly. DMs open for collabs and suggestions.
Let's get straight to it.
Everyone treats AI like a black box full of magic. It isn't. Peel it open and you find five pieces of math - most of them older than your grandparents. Once you see them, the magic never comes back. That's the point.
The smartest system humanity has ever built runs on five ideas you could fit on an index card. Not a thousand. Five: linear algebra, probability, information theory, calculus, optimization. Everything else - attention, RLHF, "emergent reasoning" - is these five wearing costumes.
I'll walk you through all five, in the exact order they fire inside a model. By the end you'll understand what's happening under the hood better than 99% of people posting about AI. And you will not be able to unsee it.
Checkpoint 01
Linear Algebra: It's All Just Matrix Multiplication
Start with the substrate. What is the model physically made of ?
Vectors and matrices. That's it. That's the whole body.
When you type a word into an AI, the first thing it does is turn that word into a list of numbers - a vector, a point floating in a space with thousands of dimensions. "King" becomes a direction. "Queen" becomes a nearby direction. Meaning stops being a mystery and becomes geometry: things that mean similar things point similar ways.
This is where it gets eerie. In 2013, a Google team led by Tomáš Mikolov (word2vec) showed you can do arithmetic on meaning itself:
vector("king") − vector("man") + vector("woman") ≈ vector("queen")
Subtract "man," add "woman," and the geometry lands you on royalty of the opposite gender. The model never learned the concept of gender. It learned a direction in space, and that direction happens to encode it. Meaning is a vector. Let that sit.
Now, every single layer of a neural network is one operation repeated:
y = Wx + b
Take your vector x, multiply it by a giant grid of numbers W (the weights), add a shift b. That's a layer. Stack a hundred of them and you have a "deep" network. The word deep just means a lot of matrix multiplications in a row. There is no deeper meaning to "deep."
Even attention - the mechanism that made ChatGPT possible, from the 2017 paper "Attention Is All You Need" - is fundamentally dot products. Here's the actual formula running inside every GPT, Claude, and Gemini alive:
Ignore the symbols and look at the core: QKᵀ. That's a pile of dot products - every word measuring how much it "aligns" with every other word. A dot product is just multiply-and-add. When the model decides that "it" refers to "the dog" three sentences back, that decision is a number produced by multiplying two lists together. That's the whole trick.
Here's single-head attention - the beating heart of the entire transformer - in eight lines:
That's not a simplification of attention. That is attention. A frontier model has more heads and bigger matrices, but the math on the screen is the math in the machine.
Everyone thinks AI "reasons." Under the hood, it's overwhelmingly matrix multiplication - Wx + b, billions of times a second - running on a chip (the GPU) built to do that one operation and nothing else. The entire AI boom is the story of humanity finally building hardware fast enough to multiply grids of numbers at ungodly scale. Nvidia became a three-trillion-dollar company selling matrix multipliers.
19th-century mathematicians - Cayley formalized matrix algebra in 1858 - handed us this math 150 years before we had the silicon to abuse it. We didn't invent new math for AI. We built the machine to run the old math a quintillion times.
Checkpoint 02
Probability: The Model Rolls a Weighted Die
Here's a fact that breaks people: the model does not know what it's about to say.
We imagine the AI "choosing" the next word like a person choosing carefully. That's not what happens. At every step, the model produces a probability distribution over its entire vocabulary - every possible next token, each with a percentage - and then it rolls a die loaded by those percentages.
The function that builds this loaded die is softmax:
It squishes the model's raw scores into clean probabilities that sum to 100%. And notice T - temperature, the most important knob you've never heard of.
Turn temperature toward zero and the die stops being a die: the model always grabs its single most likely token. Cold, rigid, repetitive. Turn it up and the distribution flattens - long-shot words get a real chance, the model gets "creative," and pushed too far it dissolves into nonsense. That "creativity" slider in your favorite AI tool? It is literally moving T in that equation.
Now the part that genuinely gave me chills. That temperature T isn't a metaphor borrowed from physics - it is physics. Softmax is mathematically identical to the Boltzmann distribution, the equation Ludwig Boltzmann wrote in the 1870s for how gas molecules spread across energy states at a given temperature. The logits are energies. Cooling the model to T→0 freezes it into its lowest-energy, single-answer state - exactly the way cooling a gas freezes it solid.
The chatbot in your browser runs 150-year-old thermodynamics to decide whether to say "cat" or "dog." Nobody tells you that.
Everyone thinks the model knows the answer. It doesn't. It builds a cloud of maybe, shaped by an equation from statistical mechanics, and hands you one sample from that cloud. Ask again and it might answer differently - not because it "changed its mind," but because it rolled the die again. Hallucinations aren't a glitch. They're the tail of the distribution getting picked. The dice were always there.
Checkpoint 03
Information Theory: It Compresses, It Doesn't Predict
This is the one that rewires your brain. It's also the one that made Claude Shannon a ghost haunting every AI lab on earth.
Everyone repeats the line: "language models predict the next word." True, but shallow. The deeper truth: to predict the next word well, you are forced to compress everything you've ever read. Prediction and compression aren't cousins - they're the same act, measured two ways. This isn't philosophy, it's a theorem.
The loss function that trains every LLM ever built - GPT, Claude, Gemini - is cross-entropy:
In plain words: it measures, in bits, how surprised the model is by the word that actually came next. Guessed it confidently? Tiny loss. Blindsided? Huge loss. Training is one long campaign to be less and less surprised by reality. A trillion-dollar industry optimizing a single number: surprise, in bits.
And here's where the ghost walks in. That equation - entropy, H = −Σ p log p, surprise in bits - was written by Claude Shannon in 1948, in a paper called "A Mathematical Theory of Communication." He introduced the bit to the world in that paper. He was trying to send phone calls down copper wire more efficiently. He had no idea he was writing the loss function for a machine that would, 78 years later, write poetry.
Then in 1951, Shannon ran an experiment identical in spirit to how we test language models today. He sat people down - his wife Betty among them - and had them guess the next letter in English text, over and over. From how often they guessed right, he computed the "entropy of English": roughly 1 bit per letter, against the ~4.7 bits you'd need if letters were random. That gap - the redundancy of language, the predictable part - is exactly what every LLM feeds on. Shannon was benchmarking a language model in 1951. The only thing missing was the computer.
That number - surprise in bits - is the only thing standing between a fresh random network and GPT-5. Drive it down, and intelligence falls out. In 2023, DeepMind made the equivalence undeniable: they used a language model as a file compressor, and it crushed the specialized tools - squeezing images tighter than PNG and audio tighter than FLAC. A model built to "predict text" out-compressed codecs built for one job over decades. Because prediction is compression. Shannon knew in 1948. We're just catching up.
Everyone thinks LLMs predict the next word. They compress language. The loss is measured in bits, the equation is 78 years old, and - the part that gives me goosebumps - Anthropic named their model Claude after Claude Shannon. The man who wrote the loss function got the model named after him. Once you see this, you cannot unsee it.
Checkpoint 04
Calculus: How It Actually Learns
So the model is made of matrices (01), outputs a weighted die (02), and is scored on its surprise in bits (03). Now the real question: how does a pile of random numbers become smart?
One question, asked a trillion times: "If I nudge this one weight a hair, does the surprise go up or down?"
That question is a derivative. And the trick that answers it for all billion weights at once is backpropagation - nothing but the chain rule from freshman calculus, run backwards through the network.
Read it as a chain of blame. The loss L was too high. Whose fault? Trace the blame backwards - how much did the output move the loss, how much did this weight move the output - multiply the links, and you get a single number: exactly how much this weight is responsible for the error. Do that for every weight in one backward sweep, and you know precisely which knobs to turn and how hard.
Here's the punchline that took me years to appreciate: there is no "learning algorithm" cleverer than this. When you hear a model was "trained for three months on 20,000 GPUs," what happened is this loop, over and over: guess, measure the surprise, run the chain rule backwards, nudge every weight a hair. Repeat a few quadrillion times. That's it. The most expensive computations in human history are freshman calculus on a loop.
And you don't even have to trust me - you can prove the framework is "just" the chain rule with your own hands. Compute the gradient manually, then let PyTorch do it, and watch the numbers match to the decimal:
loss.backward() - the single line at the center of the entire AI industry - does the exact hand-calculation above, scaled to a trillion parameters. The "magic" of a neural network learning is Leibniz's 350-year-old chain rule, plus a bookkeeping trick from 1970 (Seppo Linnainmaa) for computing all derivatives in one pass, plus a warehouse of GPUs.
Everyone pictures neural nets learning like brains. They don't. Brains don't run the chain rule backwards; nobody has found the biological wiring for it. What you're watching when an AI "learns" is calculus from the 1600s, computed in reverse, a trillion times a second. Less romantic than a digital brain. Far more terrifying, because it actually works.
Checkpoint 05
Optimization: A Ball Rolling Downhill
Backprop tells you which way is downhill for every weight. Optimization actually takes the step. And it's the least mystical idea in the whole stack: roll downhill.
∇L is the gradient - the direction the error increases fastest. So you go the opposite way (that minus sign), by a step size η (the "learning rate"). Check the slope, step downhill, check again. That's how you train an intelligence: drop a marble onto a loss landscape with a trillion dimensions and let it find the bottom.
The learning rate η is the whole art. Too big and the marble flies across the valley and off a cliff - the loss explodes to infinity, months of training gone. Too small and it crawls, and your $100M run never finishes. Every serious lab employs people whose main job is, essentially, choosing how big the steps should be.
And this idea - check the slope, step downhill - was written by Augustin-Louis Cauchy in 1847. Before electricity was wired into a single home. The optimizer training GPT-5 is a 179-year-old idea, barely changed. The modern workhorse, Adam (2014), is Cauchy's downhill step with two upgrades: momentum, so the marble builds speed through flat spots, and a per-weight step size, so each parameter tunes its own pace. Adam is one of the most-cited scientific papers in all of history - north of 100,000 citations - and it's essentially "roll downhill, but smarter."
Here's the counterintuitive kicker researchers found: in a landscape with a billion dimensions, the marble almost never gets stuck. Your intuition says a bumpy surface is full of little traps (local minima). But in ultra-high dimensions, almost every place the slope flattens is a saddle point - downhill in some direction - so there's nearly always an escape. Training works at all because of a fact about high-dimensional geometry no human can picture. The marble is rolling through a space our brains literally cannot enter.
Everyone thinks training an AI is exotic sorcery. It's a ball rolling downhill. An 1847 idea, run on a hill with a trillion dimensions instead of two. The magic isn't the method. It's the size of the mountain.
The Whole Machine, In One Breath
Now watch the five snap together into a single heartbeat - because a training step is literally these five, in order:
The model turns your text into vectors and multiplies them through its matrices - linear algebra (01). Out comes a probability distribution over the next token - probability (02). We compare that guess to reality and measure the surprise in bits - information theory (03). We run that surprise backwards through the network with the chain rule to blame every weight - calculus (04). And we nudge every weight a hair downhill - optimization (05). Then again. And again. A few quadrillion times.
That's the whole thing. A machine made of matrices, outputting dice rolls, scored on surprise, corrected by calculus, dragged downhill by an 1847 equation. Every "emergent capability," every headline, every existential debate - it all rides on top of these five moves.
The Line You Can't Uncross
Seeing these five pieces doesn't make AI "simple," and it won't settle whether the thing truly understands anything - but it does take the fear out. The moment AI stops being a glowing oracle and becomes matrix multiplication, a weighted die, Shannon's bits, the chain rule, and a marble rolling downhill, you stop being a spectator and start being someone who can see the machine underneath the magic.
And once you've seen it, you can't unsee it. Most people will spend the next decade either worshipping AI as a god or writing it off as a trick, because they never once looked inside - you just did. So the real question is this: now that you've seen the most powerful intelligence we've ever built runs on five ideas older than the car you drive, does the AI feel smaller to you, or does the math underneath it feel far larger than anyone ever taught you it was?
// They sell you the mystery. I hand you the equations. @_yusufknl //