hgfedcba:

However I do not understand your point about target bias. What exactly is target bias and how do you come up with the exact number of 25?


Your final linear layer does computation of form dot(h,w)+b, with w and b initialized to zero-centered noise, it is very far from true function f_x(x)=P(x)+25, so this offset may be modeled through h coming from earlier layers. But h is obtained via non-linear transform, so approximations suffer somewhat.



 hgfedcba:

I ran my code with all your chnges a few times and it always looks like the loss reaches a plateau at around 1 and stays there for a long time, then suddenly it gets much better. The length of this plateau varies between 200 and 800 epochs in my tests. Do you have an idea why this is? Especially when it stayed really constant at loss 1 for 800 epochs I surely would have stopped the net if it wasn’t so fast.


Not sure, it seems that curve fitting starts late, network has to shift things to a good region first. lr scheduling may also play a role.
Note that tanh is actually problematic, it has tiny gradients outside -5…5, and you’re using non-normalized inputs. So another possible reason is that some “neurons” are effectively disabled for a long time.