I’ve bumped into the same thing. It really bothered me. I should be able to counteract this mean over batch using larger learning rate, but it didn’t seem to work. Setting too high LR for Adam just blow the training. Setting it to roughly the higher value before blowing training gives just blob of white pixels in the center.
BUT then I found this (https://github.com/pytorch/examples/issues/234 124), you can read there that binary cross entropy does mean over all dimensions (batch and spatial), so reconstruction loss gets divided by much larger value then KL loss divided only by batch size. This seems to be the problem, because KL loss averaged just over batch is much larger, then reconstruction loss averaged over batch AND spatial dimension. That’s why you just get a blob, because latent distribution overfit on standard gaussian and doesn’t carry any information to decoder. If you disable averaging and then divide by size of batch, everything works well with higher LR as expected  Hope I helped!