maweibest:

I used 3 layers with about 500 neurons in each layer, and Relu activation in all layers. It seems to improve very little when I increase the model complexity.


That’s a lot actually. Get nn.Linear(3, 40) (which is same as 40 independent regressors) working and incrementally add hidden layers with a few number of units. I don’t expect MLPs to ‘just work’.