Turns out that my activation layers with inplace=True were causing the problem.
It appears that retain_graph does not work as expected with inplace operations, even if it’s just ReLU, which usually has no issues with autograd.
It would be nice if this were noted somewhere in the documentation, since the error output is misleading.