Alright, thanks to both of you for the replys!  and Zeroing the bias does help now that it’s working!
Problem was that my MNIST data is 0-1 and PIL Image was expecting 0-255 so all the data was small.  Multiplying input data by 255 makes it work.  I still do not understand why this specific problem would work/not work because of bias, seems like if small data was a problem it would also not work without the bias term.  But at least it’s working, so thanks again!