Answers.Body: <p>&quot;Piecewise linear&quot; appears to describe ReLu and LRelu activation functions, whose gradients are just simple step functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T10:35:59.863Z
Answers.LastActivityDate: 2021-08-02T10:35:59.863Z
Answers.UserDisplayName: MSalters
Body: <p>Consider the following statement (from the paper <a href="https://arxiv.org/pdf/1406.2661.pdf" rel="nofollow noreferrer">Generative Adversarial Nets</a>) about the success of discriminative models</p>
  <blockquote>
  <p>So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label. These striking successes have primarily been based on the backpropagation and dropout algorithms, <strong>using piecewise linear units which have a particularly well-behaved gradient</strong>.</p>
  </blockquote>
  <p>The piece-wise linear units they are referring to are, I guess, the activation functions. The primary purpose of activation functions is to introduce non-linearity, and there are no other mathematical requirements, such as continuity, differentiability, etc. But not all the activation functions may work well, and some are preferred over others, based on their nature, as well as the task under consideration.</p>
  <p>After reading the quoted paragraph, one can conclude that the activation functions that have <strong>well-behaved gradient</strong> are showing better results than the others, at least in discriminative tasks.</p>
  <p>What does &quot;well-behaved&quot; in this context stand for? Can we have some mathematical properties of the gradient in order to recognize it as a well-behaved gradient? Or the usage of the phrase &quot;well-behaved&quot; is highly dependent on the discriminative task under consideration?</p>
  
Tags: <deep-learning><terminology><papers><activation-functions><gradient>
Title: What is meant by "well-behaved gradient" in this context?
UserDisplayName: hanugm
ThreadText: # What is meant by "well-behaved gradient" in this context?
  
  (Asked by: username_0 on 2021-08-01)
  
  Consider the following statement (from the paper [Generative Adversarial Nets](https://arxiv.org/pdf/1406.2661.pdf)) about the success of discriminative models
  
  > So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label. These striking successes have primarily been based on the backpropagation and dropout algorithms, **using piecewise linear units which have a particularly well-behaved gradient**.
  
  The piece-wise linear units they are referring to are, I guess, the activation functions. The primary purpose of activation functions is to introduce non-linearity, and there are no other mathematical requirements, such as continuity, differentiability, etc. But not all the activation functions may work well, and some are preferred over others, based on their nature, as well as the task under consideration.
  
  After reading the quoted paragraph, one can conclude that the activation functions that have **well-behaved gradient** are showing better results than the others, at least in discriminative tasks.
  
  What does "well-behaved" in this context stand for? Can we have some mathematical properties of the gradient in order to recognize it as a well-behaved gradient? Or the usage of the phrase "well-behaved" is highly dependent on the discriminative task under consideration?
  
  **Tags:** deep-learning, terminology, papers, activation-functions, gradient
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-02. Score: 1
  
  "Piecewise linear" appears to describe ReLu and LRelu activation functions, whose gradients are just simple step functions.
  
  
  ---
  
  