Autograd

Multiply a tensor by itself and sum it, with requires_grad=True set on the input, and PyTorch doesn't wait for you to ask for a gradient before it starts working. Each operation you run appends a node to a graph right then, live, as the Python executes: y.grad_fn is that graph, and it exists the moment y does, not after some separate tracing pass discovers it.

This is define-by-run, and it's worth naming the alternative it's not, because the contrast is the whole personality of the framework. A trace-once system runs your function one time to build a static program, then replays that same program on every later call. PyTorch's tape is rebuilt from scratch on every forward pass: run the same Python twice, with a branch that goes a different way the second time, and you get two different graphs, both entirely valid, because nothing was ever fixed in advance.

The tape is built by running, and consumed by backward.

import torch

x = torch.ones(3, requires_grad=True)
y = (x * x).sum()
print(y.grad_fn)     # <SumBackward0 object ...>: the tape, built as you ran
y.backward()
print(x.grad)        # tensor([2., 2., 2.])

Call y.backward() and PyTorch walks the graph from y back toward every leaf tensor, applying the chain rule at each node and accumulating the result into that leaf's .grad. By default, once that walk finishes, the graph is gone: the intermediate values backward needed are freed, because keeping them around for a second walk nobody asked for would just be wasted memory.

Call .backward() a second time on the same y without telling PyTorch to keep the graph, and it raises, because the very thing the second call needs to walk has already been discarded. That error isn't a bug to route around quietly; it's the direct, honest consequence of a design that frees what it no longer needs. Passing retain_graph=True on the first call is how you tell PyTorch you meant to walk it again.

torch.no_grad() and torch.inference_mode() both suspend the recording itself: operations inside either context don't append anything to a graph at all, which is exactly what you want once you're only running a model forward and never planning to call backward() on the result.

Whether .grad lands anywhere depends on whether a tensor is a leaf. A leaf is a tensor you created directly, with requires_grad=True set, rather than one produced by an operation on other tensors; only leaves accumulate gradients in .grad by default. And accumulate is literal: call backward() twice in a row without clearing gradients between calls, and the second call's gradients get added on top of the first's, not swapped in for them. That's exactly why every training loop in the next chapter opens with zero_grad(): skip it, and your gradients are the sum of every step you've ever run, not just the current one.

Every operation's backward behavior comes from a rule PyTorch already knows, registered once for that primitive. Sometimes that rule is the wrong one to use, numerically unstable, or the operation itself is something autograd has no way to see through: a call out to code the tracer can't inspect. torch.autograd.Function is the escape hatch. Subclass it, define a forward and a backward by hand, and you've supplied the derivative yourself instead of trusting the automatic table.

The role this plays is the same one the JAX path's custom_vjp plays on the other side of the two philosophies: a place to stand between the framework not knowing how to differentiate something and training still working. Different tape, same fix.