Aoo_Lmay:

I think i’ll give up on that and get back to making the net wider xor deeper.


Making the network bigger will likely only make the overfitting worse. One way to think about overfitting is that your network has so many parameters that it “memorizes” the training data. That is, the network has so much “memory”—in the form of parameters that it can adjust while learning—that it remembers that input x has output y, for lots of (input, output) pairs (x,y) in the training set. So it does very well on the training set. But since it has not memorized anything about the test set, it fails miserably on the test set.
So, making the network larger will only give it more room to remember the training set, and make the overfitting worse.  Every regularization technique that you tried can be thought of as a way of limiting the amount of things that the network can memorize.
The better idea may be to make the network smaller in some manner.