If I understand correctly, your kl_divergence pushes activations towards negative values indiscriminately. So, if these 4 outputs have different moments as training starts, I would expect such a behaviour.
Note that nn.Linear initial bias is not zero is pytorch, this may aggreviate your problem. Apart from that, things to try: dropout, BatchNorm1d(affine=False), leaky_relu.