Small update: I figured out, that the network makes all output values to zero - how can that be?!
Edit: Got it - The model finished with a sigmoid activation function, while the input was a non-normalized image