Extending torch.autograd.Function (Extending PyTorch — PyTorch master documentation 1) looks like a promising option to calculate the loss from the intermediate values while maintaining the graph structure.
There’s got to be an easier solution, but extending Tensor and adding a function to retrieve a numpy ndarray of the tensor’s data without detaching it from the graph might be another option.