Answers.Body: <p>From the phrasing, it seems that complicated refers to the non-convexity of the loss landscapes of neural networks. We do not have formal guarantees of convergence in general for such landscapes. This non-convexity is a property of both the function defined by the neural network, and the particular loss function we use.</p>
  <p>In practice though, <strong>non-convexity stems from the non-linear activation functions</strong> as we almost exclusively use cross-entropy loss when training neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-22T09:28:08.697Z
Answers.LastActivityDate: 2021-08-22T09:28:08.697Z
Answers.UserDisplayName: SpiderRico
Body: <p>Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.</p>
  <p>Activation functions are a part of neural network function and neural network functions may be a part of loss functions.</p>
  <p>Consider the following paragraphs from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> of a deep learning book</p>
  <blockquote>
  <p>Optimization algorithms that use only the gradient, such as gradient
  descent, are called ﬁrst-order optimization algorithms. Optimization
  algorithms that also use the Hessian matrix, such as Newton’s method,
  are called second-order optimization algorithms (Nocedal and Wright,
  2006).</p>
  <p>The optimization algorithms employed in most contexts in this book are
  applicable to a wide variety of functions but come with almost no
  guarantees. Deep learning algorithms tend to lack guarantees because
  the <strong>family of functions used in deep learning is quite complicated.</strong> In
  many other ﬁelds, the dominant approach to optimization is to design
  optimization algorithms for a limited family of functions.</p>
  </blockquote>
  <p>The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?</p>
  
Tags: <deep-learning><optimization>
Title: Which class of functions are quite complicated in deep learning?
UserDisplayName: hanugm
ThreadText: # Which class of functions are quite complicated in deep learning?
  
  (Asked by: username_0 on 2021-08-22)
  
  Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.
  
  Activation functions are a part of neural network function and neural network functions may be a part of loss functions.
  
  Consider the following paragraphs from [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of a deep learning book
  
  > Optimization algorithms that use only the gradient, such as gradient
  > descent, are called ﬁrst-order optimization algorithms. Optimization
  > algorithms that also use the Hessian matrix, such as Newton’s method,
  > are called second-order optimization algorithms (Nocedal and Wright,
  > 2006).
  >
  > The optimization algorithms employed in most contexts in this book are
  > applicable to a wide variety of functions but come with almost no
  > guarantees. Deep learning algorithms tend to lack guarantees because
  > the **family of functions used in deep learning is quite complicated.** In
  > many other ﬁelds, the dominant approach to optimization is to design
  > optimization algorithms for a limited family of functions.
  
  The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?
  
  **Tags:** deep-learning, optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-22. Score: 1
  
  From the phrasing, it seems that complicated refers to the non-convexity of the loss landscapes of neural networks. We do not have formal guarantees of convergence in general for such landscapes. This non-convexity is a property of both the function defined by the neural network, and the particular loss function we use.
  
  In practice though, **non-convexity stems from the non-linear activation functions** as we almost exclusively use cross-entropy loss when training neural networks.
  
  
  ---
  
  