@jpeg729, you’re missing optimizer.zero_grad() at the start of the training loop. The results will change quite a bit if you add it.
python xor.py 18 --loss MSELoss --learning_rate 0.001 --activation Tanh --optimizer Adam
100 / 100 = 100.00% successes
python xor.py 18 --loss MSELoss --learning_rate 0.001 --activation Sigmoid --optimizer Adam
99 / 100 = 99.00% successes
python xor.py 18 --loss MSELoss --learning_rate 0.001 --activation ReLU --optimizer Adam
77 / 100 = 77.00% successes