Unfortunately, I cannot be really useful here, as I’m not a GAN expert. 
Generally, I would recommend to take a look at similar architectures of conditional GANs and try to have a look at the tricks, which were used to make the training converge.