So I finally figured out, that my network is just not learning at all. Changing the learning rate has zero impact and by setting a correct random seed I get the exact same results whatever learning rate I choose. After having figured that out, can anybody help me finding where I messed up the implementation?