The point is that the problem persists even after removing batchnorm. Additionally,  I’m using batch size of size 16 in my recent efforts . The same model is working under theano/lasagna implementation!