Thank you for your reply. I am doing this test before starting the training, so the learning rate doesn’t come intro play at all here. I do indeed initialize the hidden layer as zeros, but why does it work with 500 inputs but not with 3426? But again, I am not doing backpropagation here, I am just testing the untrained model.