Thanks for pointing out that final relu. As this linear test case approximation implies a strictly positive approximation function, I thought it might make sense to add this relu. Actually I ran into zero approximations sometimes. Leaving out this relu helped on these shortcomings. I was not aware of the implied zero grad.
It also allows to increase the learning rate a little but unfortunately does not tackle the core problem. I cannot find any markable difference in precision.