Answers.Body: <p>I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the <a href="https://en.wikipedia.org/wiki/Huber_loss" rel="nofollow noreferrer">Huber loss</a>(in <a href="https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html#smoothl1loss" rel="nofollow noreferrer">Pytorch</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/Huber" rel="nofollow noreferrer">TensorFlow</a>). Examples that are far from the expected value won't produce big gradients (:</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T19:29:30.977Z
Answers.Text: Yes that's correct. Thank, I will check that out
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-05T14:52:47.657Z
Answers.Text: Can I just make the gradient 0 if the error is too large? So I have 2 loss funtions, depending on the error. One is zero and the other one is quadratic or whatever else
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-06T15:08:10.033Z
Answers.Text: If you implement the loss function to be a constant, when the error is too large, you’ll get this behavior. The best way to achieve it is using torch.where/tf.where. Note that this might not be optimal. If many examples have an error larger than the threshold, the model won’t train or it will overfit on a subset of the examples. 
  Can you provide more context?
Answers.UserDisplayName: dbalchev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-06T16:40:46.467Z
Answers.Text: I'm trying to predict stock prices so the idea is after an initial phase of learning, when no error is very large and the network has a decent grasp of the function, to ditch the worst outputs in order to get better accuracy on the good outputs
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T19:25:54.827Z
Answers.LastActivityDate: 2020-09-03T19:25:54.827Z
Answers.UserDisplayName: dbalchev
Answers.Body: <p>Use genetic algorithm. Run like 25 neural networks at once and choose the most successful  one. This method is similar to evolution, which is why it is very effective. I created a model like this with similar sized training data as yours and it reached an overall error rate of 0.06% in a second. Don’t get rid of nodes. Instead, eliminate the bad networks. However, this doesn’t produce extremely high error rates if that is what you want.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T08:57:38.193Z
Answers.Text: They don't have to be high, I just wanted to say that if error is larger than x it might as well be a gazillion, I don't care. But you're just saying to run the network many times, not necessarily at the same time?
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T15:41:35.560Z
Answers.Text: Yes. Also consider increasing or decreasing your mutation amount (random increase to weights) based on how well the networks are performing.
Answers.UserDisplayName: Kral
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-04T02:42:25.780Z
Answers.Text: This will help increase network accuracy, but doesn't address the specific question about prioritising a few correct outputs over punishing very incorrect outputs. Addressing this ultimately comes down to the loss function, as that determines how to handle bad outputs.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T03:34:02.173Z
Answers.LastActivityDate: 2020-09-03T04:06:27.830Z
Answers.LastEditDate: 2020-09-03T04:06:27.830Z
Answers.UserDisplayName: Kral
Body: <p>What I want to achieve is this: If my desired outputs are [1, 2, 3, 4] I would rather have my network produce this output:</p>
  <p>[0.99, 2.01, 999, 4.01]</p>
  <p>than say this:</p>
  <p>[0.94, 1.88, 3.12, 4.1]</p>
  <p>So I'd rather have a few very accurate outputs and the rest completely off, than have them all be decent but no more than that. My question is, is there a known way to do this? If not, would it make sense to remove the inputs that produce poor outputs, and redo the learning phase?</p>
  
Tags: <neural-networks><machine-learning>
Title: Is there a way to make my neural network discard inputs with bad results from learning?
UserDisplayName: user1477107
ThreadText: # Is there a way to make my neural network discard inputs with bad results from learning?
  
  (Asked by: username_0 on 2020-09-02)
  
  What I want to achieve is this: If my desired outputs are [1, 2, 3, 4] I would rather have my network produce this output:
  
  [0.99, 2.01, 999, 4.01]
  
  than say this:
  
  [0.94, 1.88, 3.12, 4.1]
  
  So I'd rather have a few very accurate outputs and the rest completely off, than have them all be decent but no more than that. My question is, is there a known way to do this? If not, would it make sense to remove the inputs that produce poor outputs, and redo the learning phase?
  
  **Tags:** neural-networks, machine-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-09-03. Score: 3
  
  I assume [1, 2, 3, 4] are the desired outputs for different examples for a regression task. Sound like you need a different loss function. From your description it seems you don't care how big the error is if it's bigger than some value. Try the [Huber loss](https://en.wikipedia.org/wiki/Huber_loss)(in [Pytorch](https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html#smoothl1loss) and [TensorFlow](https://www.tensorflow.org/api_docs/python/tf/keras/losses/Huber)). Examples that are far from the expected value won't produce big gradients (:
  
  > **Comments:**
  > **username_0**: Yes that's correct. Thank, I will check that out  
  > **username_0**: Can I just make the gradient 0 if the error is too large? So I have 2 loss funtions, depending on the error. One is zero and the other one is quadratic or whatever else  
  > **username_1**: If you implement the loss function to be a constant, when the error is too large, you’ll get this behavior. The best way to achieve it is using torch.where/tf.where. Note that this might not be optimal. If many examples have an error larger than the threshold, the model won’t train or it will overfit on a subset of the examples.  Can you provide more context?  
  > **username_0**: I'm trying to predict stock prices so the idea is after an initial phase of learning, when no error is very large and the network has a decent grasp of the function, to ditch the worst outputs in order to get better accuracy on the good outputs  
  
  ---
  
  ### Answer by username_2 on 2020-09-03. Score: 0
  
  Use genetic algorithm. Run like 25 neural networks at once and choose the most successful one. This method is similar to evolution, which is why it is very effective. I created a model like this with similar sized training data as yours and it reached an overall error rate of 0.06% in a second. Don’t get rid of nodes. Instead, eliminate the bad networks. However, this doesn’t produce extremely high error rates if that is what you want.
  
  > **Comments:**
  > **username_0**: They don't have to be high, I just wanted to say that if error is larger than x it might as well be a gazillion, I don't care. But you're just saying to run the network many times, not necessarily at the same time?  
  > **username_2**: Yes. Also consider increasing or decreasing your mutation amount (random increase to weights) based on how well the networks are performing.  
  > **username_3**: This will help increase network accuracy, but doesn't address the specific question about prioritising a few correct outputs over punishing very incorrect outputs. Addressing this ultimately comes down to the loss function, as that determines how to handle bad outputs.  
  
  ---
  
  