Hi
You were definitly correct that RELU was not an optimal choice. I used this code
activation_function = nn.SELU()
net = nn.Sequential(nn.Linear(1, 1000), activation_function, nn.Linear(1000, 1000), activation_function, nn.Linear(1000, 1000), activation_function, nn.Linear(1000, 1))
and plugged in all 20 non-linear activation functions from here https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity 4
there were multiple where it was possible to see a curve but out of the 20 exactly one reached an somewhatdecent precision and that was SELU. However even the best activation function retained a loss of >= 1 regardless of the number of iterations, which I consider unacceptable given the task to approximate a quadratic function with 2 hidden layer of 1000 neurons.
Do you have another idea that might help? I tried different optimizers as well, but Adam + Rprop are the two that work best, so I can’t make an improvement there…
@your note: You mean I should have 2 input values, one is x and the other is x*x? I don’t get why this is better but I do believe you.