Answers.Body: <p>As long as you are arbitrarily scaling, you might try -10 to 10 or -2 to 2.  Purists will insist on values between 0 and 1, but I'd try something that doesn't terminate where x^2 == x (meaning 1).</p>
  <p>If you were using Tensorflow and Adam optimizer, I'd recommend adjusting the learning rate or LR.  It is quite possible that the Adam optimizer is available in pytorch and you can find LR either in the documentation or by typing a dot and hitting tab.</p>
  <p>All that said, I'd start with 4 layers of 100 neurons and only use relu activation.  It seems a little extreme, but it worked for me when first &quot;breaking linearity&quot; (in TF).</p>
  <p>Finally, I have found that the number of epochs for a good fit is a real matter of feel.  10, 100, 1000, anything really, when you're feeling around for results.  My gut says 100.</p>
  <p>Normally, I wouldn't add my newb $0.02, but it's been 3.14 months.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T19:53:38.963Z
Answers.LastActivityDate: 2024-11-29T19:53:38.963Z
Answers.UserDisplayName: James Huddle
Body: <p>I'm aghast at how difficult of a problem it is for me to solve the function <code>f(x) = x^2</code> with a linear-regression multi-layer-perceptron approach with PyTorch.</p>
  <p>I'm using SGD for the optimizer, MSELoss for the loss-function, and setting up a single hidden-layer with eight hidden nodes.</p>
  <p>I'm using ReLU as the nonlinear activation function between layers.</p>
  <p>Further, I'm scaling the input--arbitrarily--between -1000 and 1000.</p>
  <p>My rescaling logic is sound; it <em>is</em> possible I've err'd; however, I used the same logic for rescaling my inputs and outputs for <code>f(x) = 2x</code> and it was working.</p>
  <p>My loss curve is arbitrarily going up and down without progress; should this approach work but require more training, or am I fundamentally approaching this problem incorrectly?</p>
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-21T14:33:28.957Z
Comments.Text: If your inputs take range +/- 1e3, then your outputs will be in range +/- 1e6, then mean squared error will be in range +/- 1e12. As a result, SGD is likely taking huge steps which could cause the unstable training you're observing. It's *very* common to normalize the inputs to a network, here's just one of many discussions https://stats.stackexchange.com/questions/421927/neural-networks-input-data-normalization-and-centering/422087#422087
Comments.UserDisplayName: bogovicj
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-28T18:15:15.403Z
Comments.Text: @bogovicj are there parameters to adjust SGD to lessen the step-size, and/or is there a different optimizer altogether I should be using for its--I'd guess fervently--parameter customization options?
Comments.UserDisplayName: Tyler Curtis Jowers
Tags: <machine-learning><pytorch><multilayer-perceptrons><relu>
Title: How can you solve a machine-learning linear-regression problem for a curvilinear relationship?
UserDisplayName: Tyler Curtis Jowers
ThreadText: # How can you solve a machine-learning linear-regression problem for a curvilinear relationship?
  
  (Asked by: username_0 on 2024-08-19)
  
  I'm aghast at how difficult of a problem it is for me to solve the function `f(x) = x^2` with a linear-regression multi-layer-perceptron approach with PyTorch.
  
  I'm using SGD for the optimizer, MSELoss for the loss-function, and setting up a single hidden-layer with eight hidden nodes.
  
  I'm using ReLU as the nonlinear activation function between layers.
  
  Further, I'm scaling the input--arbitrarily--between -1000 and 1000.
  
  My rescaling logic is sound; it *is* possible I've err'd; however, I used the same logic for rescaling my inputs and outputs for `f(x) = 2x` and it was working.
  
  My loss curve is arbitrarily going up and down without progress; should this approach work but require more training, or am I fundamentally approaching this problem incorrectly?
  
  Thank you.
  
  **Tags:** machine-learning, pytorch, multilayer-perceptrons, relu
  
  > **Comments:**
  > **username_1**: If your inputs take range +/- 1e3, then your outputs will be in range +/- 1e6, then mean squared error will be in range +/- 1e12. As a result, SGD is likely taking huge steps which could cause the unstable training you're observing. It's *very* common to normalize the inputs to a network, here's just one of many discussions https://stats.stackexchange.com/questions/421927/neural-networks-input-data-normalization-and-centering/422087#422087  
  > **username_0**: @username_1 are there parameters to adjust SGD to lessen the step-size, and/or is there a different optimizer altogether I should be using for its--I'd guess fervently--parameter customization options?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-29. Score: 0
  
  As long as you are arbitrarily scaling, you might try -10 to 10 or -2 to 2. Purists will insist on values between 0 and 1, but I'd try something that doesn't terminate where x^2 == x (meaning 1).
  
  If you were using Tensorflow and Adam optimizer, I'd recommend adjusting the learning rate or LR. It is quite possible that the Adam optimizer is available in pytorch and you can find LR either in the documentation or by typing a dot and hitting tab.
  
  All that said, I'd start with 4 layers of 100 neurons and only use relu activation. It seems a little extreme, but it worked for me when first "breaking linearity" (in TF).
  
  Finally, I have found that the number of epochs for a good fit is a real matter of feel. 10, 100, 1000, anything really, when you're feeling around for results. My gut says 100.
  
  Normally, I wouldn't add my newb $0.02, but it's been 3.14 months.
  
  
  ---
  
  