Answers.Body: <p><a href="https://en.wikipedia.org/wiki/Catastrophic_cancellation" rel="nofollow noreferrer">Catastrophic cancellation</a> occurs when a function to optimise includes the difference between two estimates to close numbers. As those estimates approach their true values, the ratio of the estimated difference between them and real difference between them may vary widely.</p>
  <p>Although this may be a concern in a particular model or problem, it is not really related to neural network loss functions or how they learn from example data. No standard loss functions rely on a stable relative difference between two estimates from the neural network. Most loss functions compare a current estimate with a target value (or ground truth), and the error value for that is absolute and can approach zero without causing any instability.</p>
  <p>There may be a few exceptions, and in addition there <em>are</em> other numerical issues that impact convergence stability for neural networks, where solutions are employed in neural network libraries. For example when dealing with output layers with exponentiation in them (e.g. sigmoid or softmax) combined with loss functions that have log terms in them (e.g. cross entropy), it is common to use a simplified gradient calculation that does not need to actually perform the exponentiation or log - this saves time and is numerically more stable. This is not the same issue as catastrophic cancellation, but clearly is considered important enough in neural networks that it is addressed.</p>
  <p>As far as I know, there is no need to address catastropic cancellation in general neural network optimisation, and there are no libraries with standard solutions.</p>
  <p>If you <em>are</em> estimating two close values, taking the difference, perhaps in some pipeline, and the resulting system is sensitive to proportional errors in that difference, then you may be concerned about catastrophic cancellation and need to do something about it. What that something is will not directly be related to neural network libraries though. Perhaps you will be able to reformulate your problem so that the difference between the estimates is not so important, or that measure could be estimated more directly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-26T20:03:40.590Z
Answers.LastActivityDate: 2022-04-26T20:03:40.590Z
Answers.UserDisplayName: Neil Slater
Body: <p>It just occurred to me that this seems like it should be a very common problem that must have some kind of solution... Yet I'm not sure what it is...</p>
  <p>If there is no solution, does this mean once a model reaches a certain level of accuracy numerical training methods can no longer be counted on to make it 'better'? And if so, is this a practical concern?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T19:04:26.230Z
Comments.Text: Could you please give an example of a "no solution" scenario to better illustrate the kind of thing you are thinking of and to make this more concrete? For instance, very often there is an optimal statistical solution to a data fitting problem. As a trivial example if you have a function $y=f(x)$ to learn and contradictory training examples $(x,y)$ of $(1, 0), (1, 1)$, then many neural networks will approximate $\hat{f}(1) = 0.5$ - it's valid and optimal according to the loss function. But not clear if this matches your "if there is no solution" scenario.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T18:33:58.033Z
Comments.Text: By 'no solution' I mean that there are no common mechanisms implemented in libraries (e.g. keras, sklearn, etc...) to counter the affects of catastrophic cancellation on the loss function when the model is already quite close to optimal.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T18:34:40.343Z
Comments.Text: Intuitively, it seems like unless this problem is addressed it might cause models which are nearly perfect to start to diverge. No?
Comments.UserDisplayName: profPlum
Tags: <neural-networks><objective-functions><numerical-algorithms>
Title: How is catastrophic cancellation dealt with in loss functions?
UserDisplayName: profPlum
ThreadText: # How is catastrophic cancellation dealt with in loss functions?
  
  (Asked by: username_0 on 2022-04-25)
  
  It just occurred to me that this seems like it should be a very common problem that must have some kind of solution... Yet I'm not sure what it is...
  
  If there is no solution, does this mean once a model reaches a certain level of accuracy numerical training methods can no longer be counted on to make it 'better'? And if so, is this a practical concern?
  
  **Tags:** neural-networks, objective-functions, numerical-algorithms
  > **Comments:**
  > **username_1**: Could you please give an example of a "no solution" scenario to better illustrate the kind of thing you are thinking of and to make this more concrete? For instance, very often there is an optimal statistical solution to a data fitting problem. As a trivial example if you have a function $y=f(x)$ to learn and contradictory training examples $(x,y)$ of $(1, 0), (1, 1)$, then many neural networks will approximate $\hat{f}(1) = 0.5$ - it's valid and optimal according to the loss function. But not clear if this matches your "if there is no solution" scenario.  
  > **username_0**: By 'no solution' I mean that there are no common mechanisms implemented in libraries (e.g. keras, sklearn, etc...) to counter the affects of catastrophic cancellation on the loss function when the model is already quite close to optimal.  
  > **username_0**: Intuitively, it seems like unless this problem is addressed it might cause models which are nearly perfect to start to diverge. No?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-26. Score: 1 | Accepted answer
  
  [Catastrophic cancellation](https://en.wikipedia.org/wiki/Catastrophic_cancellation) occurs when a function to optimise includes the difference between two estimates to close numbers. As those estimates approach their true values, the ratio of the estimated difference between them and real difference between them may vary widely.
  
  Although this may be a concern in a particular model or problem, it is not really related to neural network loss functions or how they learn from example data. No standard loss functions rely on a stable relative difference between two estimates from the neural network. Most loss functions compare a current estimate with a target value (or ground truth), and the error value for that is absolute and can approach zero without causing any instability.
  
  There may be a few exceptions, and in addition there *are* other numerical issues that impact convergence stability for neural networks, where solutions are employed in neural network libraries. For example when dealing with output layers with exponentiation in them (e.g. sigmoid or softmax) combined with loss functions that have log terms in them (e.g. cross entropy), it is common to use a simplified gradient calculation that does not need to actually perform the exponentiation or log - this saves time and is numerically more stable. This is not the same issue as catastrophic cancellation, but clearly is considered important enough in neural networks that it is addressed.
  
  As far as I know, there is no need to address catastropic cancellation in general neural network optimisation, and there are no libraries with standard solutions.
  
  If you *are* estimating two close values, taking the difference, perhaps in some pipeline, and the resulting system is sensitive to proportional errors in that difference, then you may be concerned about catastrophic cancellation and need to do something about it. What that something is will not directly be related to neural network libraries though. Perhaps you will be able to reformulate your problem so that the difference between the estimates is not so important, or that measure could be estimated more directly.
  
  
  ---
  
  