Here for the update. Turns out my dataset is not the main problem (It took sometime to verfiy). The root of my problem was how I structured the network architecture. I was implementing a type of variational autoencoder. Instead of implementing it the “right” way (how it is generally implemented). I tried to cheat by partitioning 2x2 output tensor of the previous layer into 2x1 tensors first, then I summed the ensuing two tensors. Then again I partitoned the 2x1 tensor into two 1x1 tensors. I used the ensuing tensors as  “mu” and “logvar” (the mean and the variation vectors) due to memory restrictions. For small a dataset, no problem emerged that I could detect. when the dataset got larger I got the problems I mentioned here. Solution is following the convention, which is using two different fully connected layer (or I guess one can also use 1x1 convolutions) as the “mu” and “logvar” vectors.