Thanks alot, I will look into that. I changed the optimizer to Adam and tried to run it for more epochs and in the end it converged. But yeah, I forgot some activation functions, will fix that, guess that will improve things.