Do you see an unusual long duration for freezing / unfreezing the models?
If freezing and unfreezing takes more time than just calculating the gradients, you could just remove it and clear the gradients corresponding to your use case.
As a side note: Did you forget to unfreeze the adversary at the end of your code?