~/blog/micrograd
Published on

Micrograd, Line by Line — How Andrej Karpathy's Tiny Autograd Engine Explains Neural Nets

1771 words9 min read
Authors

If you've ever wanted to actually understand what loss.backward() is doing instead of just trusting it, watch Andrej Karpathy's "The spelled-out intro to neural networks and backpropagation: building micrograd". It's the best, free, most intuitive explanation of backpropagation, forward passes, and gradient descent I've come across — and it gets there by building a working neural network from absolute scratch, in about 150 lines of Python, with zero dependencies beyond the standard library.

The code lives at karpathy/micrograd, and it's small enough to read start to finish in one sitting. This post walks through the pieces — the Value object, the forward pass, the backward pass, and the Neuron / Layer / MLP classes built on top of it — in the order they clicked for me.

Why a Toy Autograd Engine Is the Best Way In

PyTorch and TensorFlow do the same thing micrograd does, just on tensors instead of scalars, with a lot more engineering around performance, hardware, and edge cases. Strip all of that away and what's left is a genuinely small idea: every number involved in a computation can remember how it was computed, and that memory is enough to work out how much it contributed to the final result. That's it. That's automatic differentiation. Micrograd makes that idea impossible to hide behind abstractions, because you write the abstraction yourself.

The Value Object: A Number That Remembers How It Was Made

Everything starts with a wrapper around a plain float. A Value doesn't just hold a number — it holds a reference to whatever Values produced it and the operation that combined them:

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(children)
        self._op = op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')

        def _backward():
            self.grad += out.grad
            other.grad += out.grad

        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')

        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad

        out._backward = _backward
        return out

Two things worth pausing on:

  • Every operation (__add__, __mul__, and so on) returns a new Value that keeps a pointer back to its inputs. Do enough arithmetic and you've silently built a directed graph.
  • Each operation also stashes a tiny _backward closure that knows the local derivative for that specific operation. Addition just passes the gradient straight through to both parents; multiplication scales it by the other input. Neither closure needs to know anything about the rest of the graph — that's the whole trick.

Forward Pass: Just Python, Secretly Building a Graph

Because + and * are overloaded, writing the forward pass of a tiny network looks exactly like normal math:

x1, x2 = Value(2.0), Value(0.0)
w1, w2 = Value(-3.0), Value(1.0)
b = Value(6.7)

n = x1 * w1 + x2 * w2 + b
o = n.tanh()

Nothing about this line looks special — but by the time o exists, it's sitting at the root of a graph that includes every intermediate multiplication and addition that produced it, each one holding onto the local derivative it'll need later. The forward pass and the graph construction happen in the same breath.

Backward Pass: The Chain Rule, Automated

Calling .backward() on the final output does two things: it seeds that node's gradient at 1.0 (the output's derivative with respect to itself), then walks the graph in reverse topological order, calling each node's _backward() closure so gradients accumulate from the output back to every leaf:

def backward(self):
    topo = []
    visited = set()

    def build(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build(child)
            topo.append(v)

    build(self)

    self.grad = 1.0
    for v in reversed(topo):
        v._backward()

That topological sort matters: a node's gradient isn't final until every node that depends on it has already propagated its own gradient backward. Sort the graph so children always come before parents, walk it in reverse, and the chain rule falls out automatically — no calculus by hand, no symbolic differentiation, just local rules composing correctly because the order is right.

Neuron, Layer, MLP: Stacking Value Math Into a Network

Once Value can add, multiply, and back-propagate through itself, an actual neural network is just bookkeeping on top of it. A neuron is a list of weights, a bias, and a nonlinearity:

class Neuron:
    def __init__(self, n_inputs):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)]
        self.b = Value(0.0)

    def __call__(self, x):
        act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()

    def parameters(self):
        return self.w + [self.b]

A layer is just a handful of neurons that all see the same input:

class Layer:
    def __init__(self, n_inputs, n_outputs):
        self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)]

    def __call__(self, x):
        outs = [n(x) for n in self.neurons]
        return outs[0] if len(outs) == 1 else outs

    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

And an MLP is just layers feeding into each other:

class MLP:
    def __init__(self, n_inputs, layer_sizes):
        sizes = [n_inputs] + layer_sizes
        self.layers = [Layer(sizes[i], sizes[i + 1]) for i in range(len(layer_sizes))]

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

There's no magic left by this point — MLP is a Layer of Layers of Neurons, and every single number flowing through it is a Value quietly building the graph it'll need for .backward().

Gradient Descent, in About Ten Lines

With a model and a loss function, training is a loop: predict, measure how wrong you were, propagate that wrongness backward, and nudge every parameter a little in the direction that reduces it.

model = MLP(3, [4, 4, 1])

for step in range(50):
    y_pred = [model(x) for x in xs]
    loss = sum((yp - yt) ** 2 for yt, yp in zip(ys, y_pred))

    for p in model.parameters():
        p.grad = 0.0
    loss.backward()

    for p in model.parameters():
        p.data += -0.05 * p.grad

Zero the gradients first (they accumulate, so stale gradients from the last step would otherwise leak in), backward the loss to fill every parameter's .grad, then step each parameter opposite its gradient, scaled by a learning rate. Run that loop enough times and the loss goes down. That's gradient descent — the same algorithm training billion-parameter models, just visible end to end.

Why It's Worth Your Time

I'd used autograd for a while before watching this without really being able to explain what it was doing under the hood. Micrograd doesn't hide behind tensors, GPUs, or a huge framework — it's small enough to hold in your head completely, and once you do, PyTorch's .backward() stops feeling like magic and starts feeling like exactly the code above, scaled up. If you're working with neural nets and haven't sat down with this yet, it's a couple of hours genuinely well spent.