Library > karpathy/micrograd

Micrograd is a minimalist scalar-valued automatic differentiation engine accompanied by a small neural network library, created by Andrej Karpathy as an educational project. The core of the library is the Value class in engine.py, which wraps scalar numbers, overloads Python arithmetic operators, and builds a dynamic computation graph during the forward pass. Reverse-mode autodiff (backpropagation) is then performed by calling .backward(), propagating gradients through the graph using closures stored at each node. Built on top of this engine, nn.py provides three composable abstractions — Neuron, Layer, and MLP — that mirror the structure of real deep learning frameworks like PyTorch but in only a few dozen lines of code. This deliberate minimalism is the key design decision: every concept (gradient accumulation, topological sort for backprop, parameter management) is visible and readable without abstraction overhead. The primary audience is learners who want to deeply understand how neural networks and backpropagation work under the hood. It is also useful as a reference implementation or starting point for building more complex autograd systems. The test suite validates gradient correctness by comparing results against PyTorch, making it easy to verify that the tiny engine behaves correctly.

Start Reading →Browse All Files

Reading Guide

1
The Value Class: Wrapping Scalars in a Computation Graph
This chapter introduces the foundational data structure of the entire library: the Value class in engine.py. You will learn how a simple Python class wraps a scalar number, records its history of operations, and lays the groundwork for automatic differentiation. This is the single most important file in the repository — everything else depends on understanding it.
2
Backpropagation: Topological Sort and Gradient Flow
With the computation graph built during the forward pass, this chapter explains how gradients are computed in reverse order via the .backward() method. You will learn why a topological sort is necessary, how the gradient of the final output is seeded, and how the chain rule is applied automatically across the entire graph.
3
Neural Network Abstractions: Neuron, Layer, and MLP
With the autograd engine in place, nn.py builds three composable neural network abstractions on top of it. This chapter walks through how weights and biases are represented as Value objects, how the forward pass of a neuron is computed, and how these primitives compose into full multi-layer perceptrons.
4
Testing: Verifying Correctness Against PyTorch
The test suite validates micrograd's gradient computations by comparing them against PyTorch as a ground-truth oracle. This chapter explains the testing strategy, what the two test functions cover, and what edge cases they expose about the autograd engine's correctness.
5
Package Structure and the Public API
This short chapter covers the packaging layer of micrograd: the __init__.py that defines the public namespace and setup.py that configures distribution. While not essential to understanding the autograd engine, these files reveal how the project is intended to be used and installed as a proper Python package.

Architecture

Micrograd/Root/Test
Micrograd is a minimal scalar-valued autograd engine with a neural network library built on top, enabling automatic differentiation and backpropagation for training simple neural networks.

Entry Points

micrograd/engine.pymicrograd/nn.py