After the backward() call on the output of the classifier network, the parameters of the diffusion network will get their gradients populated (or accumulated if the gradients are already set to some value). The classifier network just adds operations on top of the first network.
This sort of behaviour can be easily disabled by calling detach() on the output of the first network, before passing it as an input to the second network.
output_classifier = classifier(output_diffusion.detach())

detach() returns a new tensor that points to the same underlying storage, but no longer tracks gradient (requires_grad is set to False). This means that the gradient that flows through the second network can no longer be propageted through the first network.