With relu’s and learning rate = 0.1 will often converge to the solution, but will often also converge to a degenerate solution where it assigns the same probability to both labels.
Using tanh instead of relu seems to work better in this case.