I’m still quite confused… So you are not using non-linearity at all? Wouldn’t that be just a linear model?
How large did you set the hidden layer size? Even with standard normal initialization, i.e. initial output is a normal centered at 0 with linear model, it shouldn’t easily go beyond double range. Double range is very very large.
Also, you may want to initialize using some heuristic depending on the input (and output) size, such as randn(n) / sqrt(n) (n is input size), etc.
If Chinese is easier for your to explain this, feel free to message me in Chinese.