Your loss will be a bit higher, since you pass reduction='sum' to nn.CrossEntropyLoss.
Would the default elementwise_mean not work?
EDIT: That being said, you should also have a look at Conditional GANs 8, which use a slightly different approach to create class-specific images.