Please elaborate on what do you exactly mean by not converging: do you mean that your trainging loss goes down while your validation loss goes up, or that both of your losses do not always go down to the same value in the same number of batches?
Also, are you training “usual” network, or a gan?