Should I be using batchnorm and/or dropout in a VAE or GAN? I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm.
Here are my thoughts (I may be completely wrong):
Dropout:
From my understanding, this is used in supervised networks to curb overfitting by making sure each neuron represents something meaningful, rather than memorizing the training data.
For GANs, my guess is that dropout can be used to prevent the discriminator or generator from being too strong, therefore helping reduce the chance of mode collapse?
For VAE, I don't think dropout is useful? Not sure.
Batch norm:
From my understanding, batch norm reduces covariate shift inside of a neural network, which can be observed when you have different training and testing distributions.
Therefore, I think this isn't really applicable in GANs, since at test time we simply sample from a pre-set distribution (commonly used is $\mathcal{N}(0,1)$), the input data is usually from the same distribution.
On the other hand VAE can be affected by covariate shift, since the inputs at test time may have a different distribution than the training set.

Dropout: I agree with comments saying that dropout has mostly been dropped (ha) in favor of other regularization techniques, especially as architectures have gone more fully convolutional (and dropout doesn't really work with conv layers). Also note that dropout and batch norm can have bad interactions with each other.
I don't think anyone really understands why batch norm helps - some have argued against "covariate shift" for example - How Does Batch Normalization Help Optimization?. So I don't this is a strike against using it in GANs.
Some GAN varieties like WGAN assume independence between samples in a batch, which is a good reason to avoid batch norm.
