I think 0.001 is pretty standard value, so it’s weird for me that it doesn’t work at all. Yes, the same with 0.0001.
Also I think the network is very sensitive to the SGD learning rate. It barely works with any changes