OK, adding create_graph=True to the first backward call fixes the problem. From the docs:

create_graph (bool, optional) – If True, graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults to False.

I kind of expected a warning or error, similar to when calling backward twice (without explicitly retaining the graph), but I guess it might be hard to detect if cutting the computational graph at that point is intended or not…