I found out the issue. I am using this Face Alignment 5 model to align the face images and it turns out that internally it calls torch.set_grad_enabled(False) which apparently turns off the gradient computation globally in PyTorch.
To solve the issue I just call torch.set_grad_enabled(True) when training the GAN.