Answers.Body: <p>The maximum function is not smooth, since it's first derivative is not continuous.</p>
  <p>Having non-smooth functions is generally a bad thing for neural networks, since they don't work nicely with gradient decent.</p>
  <p>So what you want is a smooth approximation to these functions.</p>
  <p>Logsumexp is the smooth approximation to the maximum function and so it is what you should use in a neural network, just like softmax is a smooth approximation to the argmax
  <a href="https://en.wikipedia.org/wiki/LogSumExp" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/LogSumExp</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T20:19:20.083Z
Answers.Text: People have been using ReLu functions just fine. Differentiable almost everywhere is probably fine for most applications.
Answers.UserDisplayName: Stefan Perko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-15T09:45:02.813Z
Answers.Text: @StefanPerko `SoftPlus(x) = log(1 + exp(x))` is a smooth approximation to relu which unlike relu would allow the gradients to propagate backwards for x<0, but I doubt there would be any benefit since gradient decent doesn't get stuck with relu anyway, which is interesting
Answers.UserDisplayName: Tom Huntington
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-12T03:41:01.560Z
Answers.LastActivityDate: 2022-05-15T09:33:08.443Z
Answers.LastEditDate: 2022-05-15T09:33:08.443Z
Answers.UserDisplayName: Tom Huntington
Body: <p>I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.</p>
  <p>Is there some kind of &quot;weighted maximum&quot; that allows the gradients to backpropagate?</p>
  <p>Edit: I had a two dimensional tensor (correlation matrix) I wanted to reduce to one dimension.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-13T08:59:48.723Z
Comments.Text: "wanting to add a maximum in my neural network", do you mean as an activation function? But how? It's not clear where you would use this "max" function. Edit your post to clarify that.
Comments.UserDisplayName: nbro
Tags: <neural-networks><backpropagation>
Title: Is there some kind of "weighted maximum" that allows the gradients to backpropagate?
UserDisplayName: Tom Huntington
ThreadText: # Is there some kind of "weighted maximum" that allows the gradients to backpropagate?
  
  (Asked by: username_0 on 2022-05-12)
  
  I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.
  
  Is there some kind of "weighted maximum" that allows the gradients to backpropagate?
  
  Edit: I had a two dimensional tensor (correlation matrix) I wanted to reduce to one dimension.
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_1**: "wanting to add a maximum in my neural network", do you mean as an activation function? But how? It's not clear where you would use this "max" function. Edit your post to clarify that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-05-12. Score: 1 | Accepted answer
  
  The maximum function is not smooth, since it's first derivative is not continuous.
  
  Having non-smooth functions is generally a bad thing for neural networks, since they don't work nicely with gradient decent.
  
  So what you want is a smooth approximation to these functions.
  
  Logsumexp is the smooth approximation to the maximum function and so it is what you should use in a neural network, just like softmax is a smooth approximation to the argmax
  <https://en.wikipedia.org/wiki/LogSumExp>
  
  > **Comments:**
  > **username_2**: People have been using ReLu functions just fine. Differentiable almost everywhere is probably fine for most applications.  
  > **username_0**: @username_2 `SoftPlus(x) = log(1 + exp(x))` is a smooth approximation to relu which unlike relu would allow the gradients to propagate backwards for x<0, but I doubt there would be any benefit since gradient decent doesn't get stuck with relu anyway, which is interesting  
  
  ---
  
  