Answers.Body: <p>A couple things you could try:</p>
  <p>You could try normalizing your target variable, so that it's number of standard deviations from the mean, or mapped to [-1,1].</p>
  <p>If you are using drop-out during training. Make sure you turn it off during inference time.</p>
  <p>Verify something isn't funky with your test set. Combine your training and test set and re-split it. Are you seeing the same behavior?</p>
  <p>1 epoch isn't much, what happens when you continue training?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-04T16:32:56.523Z
Answers.Text: Yes, my targets values range from -1 to 1, the average loss in epoch 2 just stayed around 0.089 with slightly drecreasing,not fast like epoch 1 so I stopped it, I also use model.eval() when predicting so I assume that dropout is not a cause.
Answers.UserDisplayName: Việt Nguyễn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T06:47:03.927Z
Answers.Text: What happens when you resplit your dataset? Same thing? Are you certain you are doing the same preprocessing on the train and test set?
Answers.UserDisplayName: Michael Higgins
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-09T02:30:46.080Z
Answers.Text: same preprocessing because they are just texts, so I just clean some url links and stop words
Answers.UserDisplayName: Việt Nguyễn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-09T02:38:39.867Z
Answers.Text: But I really change the structure of label and input. This task  actually is about order ranking. I calculate relative rank of one item compared with the others by the formulas: d(rank_i,rank_j) = rank_i -rank_j, so if I want to know the order of rank_j relative to all of item i’s in dataset, I just need to subtract and find all the d’s.That’s why my dataset labels have a lot of negative and positive values and it ranges from -1 to 1. The problem may be I’ve changed the structure and the model does not see any pattern in it?
Answers.UserDisplayName: Việt Nguyễn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T08:56:22.453Z
Answers.Text: Where are you getting ground truth for the ranks? When you say relative to "all of items i's in dataset" what do you mean exactly? What dataset ... train, test or both.
Answers.UserDisplayName: Michael Higgins
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-04T15:36:23.570Z
Answers.LastActivityDate: 2022-06-04T15:36:23.570Z
Answers.UserDisplayName: Michael Higgins
Body: <p>I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, the loss goes down to 0.089, but when I try to predict on test-set, every samples have the same values.</p>
  <p>I try to add tanh activation in the last layer and switch to roberta base model, this time the model predict different values, on the train set, it predicts positive and negative values but on the test-set, it only produces positive ones.</p>
  <p>Is there any way to train a regression model with negative values that is more stable?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-04T21:01:16.913Z
Comments.Text: Sounds pathological. I Would plot the data that you are using for your targets in a NIST 4-plot. A little bit of gross reality checking can save you some grief. There is pre-processing that takes things like the mean and the standard deviation away from the learner, so it can major on the hard parts. I would also look for indexing issues in your code, things where you are thinking your for loop is changing the variable “i” but the variable is only looking at “1“ or similar.
Comments.UserDisplayName: EngrStudent
Tags: <machine-learning><natural-language-processing><activation-functions><prediction><regression>
Title: Is there any way to train a regression model with negative values that is more stable?
UserDisplayName: Việt Nguyễn
ThreadText: # Is there any way to train a regression model with negative values that is more stable?
  
  (Asked by: username_0 on 2022-06-04)
  
  I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, the loss goes down to 0.089, but when I try to predict on test-set, every samples have the same values.
  
  I try to add tanh activation in the last layer and switch to roberta base model, this time the model predict different values, on the train set, it predicts positive and negative values but on the test-set, it only produces positive ones.
  
  Is there any way to train a regression model with negative values that is more stable?
  
  **Tags:** machine-learning, natural-language-processing, activation-functions, prediction, regression
  
  > **Comments:**
  > **username_1**: Sounds pathological. I Would plot the data that you are using for your targets in a NIST 4-plot. A little bit of gross reality checking can save you some grief. There is pre-processing that takes things like the mean and the standard deviation away from the learner, so it can major on the hard parts. I would also look for indexing issues in your code, things where you are thinking your for loop is changing the variable “i” but the variable is only looking at “1“ or similar.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-04. Score: 1
  
  A couple things you could try:
  
  You could try normalizing your target variable, so that it's number of standard deviations from the mean, or mapped to [-1,1].
  
  If you are using drop-out during training. Make sure you turn it off during inference time.
  
  Verify something isn't funky with your test set. Combine your training and test set and re-split it. Are you seeing the same behavior?
  
  1 epoch isn't much, what happens when you continue training?
  
  > **Comments:**
  > **username_0**: Yes, my targets values range from -1 to 1, the average loss in epoch 2 just stayed around 0.089 with slightly drecreasing,not fast like epoch 1 so I stopped it, I also use model.eval() when predicting so I assume that dropout is not a cause.  
  > **username_2**: What happens when you resplit your dataset? Same thing? Are you certain you are doing the same preprocessing on the train and test set?  
  > **username_0**: same preprocessing because they are just texts, so I just clean some url links and stop words  
  > **username_0**: But I really change the structure of label and input. This task  actually is about order ranking. I calculate relative rank of one item compared with the others by the formulas: d(rank_i,rank_j) = rank_i -rank_j, so if I want to know the order of rank_j relative to all of item i’s in dataset, I just need to subtract and find all the d’s.That’s why my dataset labels have a lot of negative and positive values and it ranges from -1 to 1. The problem may be I’ve changed the structure and the model does not see any pattern in it?  
  > **username_2**: Where are you getting ground truth for the ranks? When you say relative to "all of items i's in dataset" what do you mean exactly? What dataset ... train, test or both.  
  
  ---
  
  