Solved by spadel in post #2 
              
              
                Small update: I figured out, that the network makes all output values to zero - how can that be?! 
Edit: Got it - The model finished with a sigmoid activation function, while the input was a non-normalized image