Solved by vfdev-5 in post #4 
              
              
                As you can find in various source on how to debug CNN and training: 

try to overfit the model on a single batch

Seems like this model without BN has a difficulty even to overfit a single batch. 
I tried a larger LR -> 0.1 and loss on a single batch stucked on ~4.0 
With BN’s seems like it can over…