Btw this turned out to be a problem of dead ReLu’s. switching to LeakReLu or something else got the weight to behave little more like expected but the network would eventually die. Upon switching from Adam to SGD optimizer the network never dies anymore and learns. Not sure why that did the trick.