<p>From the phrasing, it seems that complicated refers to the non-convexity of the loss landscapes of neural networks. We do not have formal guarantees of convergence in general for such landscapes. This non-convexity is a property of both the function defined by the neural network, and the particular loss function we use.</p>
<p>In practice though, <strong>non-convexity stems from the non-linear activation functions</strong> as we almost exclusively use cross-entropy loss when training neural networks.</p>

CC BY-SA 4.0
2021-08-22T09:28:08.697Z
2021-08-22T09:28:08.697Z
SpiderRico
<p>Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.</p>
<p>Activation functions are a part of neural network function and neural network functions may be a part of loss functions.</p>
<p>Consider the following paragraphs from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> of a deep learning book</p>
<blockquote>
<p>Optimization algorithms that use only the gradient, such as gradient
descent, are called ﬁrst-order optimization algorithms. Optimization
algorithms that also use the Hessian matrix, such as Newton’s method,
are called second-order optimization algorithms (Nocedal and Wright,
2006).</p>
<p>The optimization algorithms employed in most contexts in this book are
applicable to a wide variety of functions but come with almost no
guarantees. Deep learning algorithms tend to lack guarantees because
the <strong>family of functions used in deep learning is quite complicated.</strong> In
many other ﬁelds, the dominant approach to optimization is to design
optimization algorithms for a limited family of functions.</p>
</blockquote>
<p>The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?</p>

<deep-learning><optimization>
Which class of functions are quite complicated in deep learning?
hanugm
# Which class of functions are quite complicated in deep learning?

(Asked by: username_0 on 2021-08-22)

Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.

Activation functions are a part of neural network function and neural network functions may be a part of loss functions.

Consider the following paragraphs from [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of a deep learning book

> Optimization algorithms that use only the gradient, such as gradient
> descent, are called ﬁrst-order optimization algorithms. Optimization
> algorithms that also use the Hessian matrix, such as Newton’s method,
> are called second-order optimization algorithms (Nocedal and Wright,
> 2006).
>
> The optimization algorithms employed in most contexts in this book are
> applicable to a wide variety of functions but come with almost no
> guarantees. Deep learning algorithms tend to lack guarantees because
> the **family of functions used in deep learning is quite complicated.** In
> many other ﬁelds, the dominant approach to optimization is to design
> optimization algorithms for a limited family of functions.

The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?

**Tags:** deep-learning, optimization


---

## 1 answers

### Answer by username_1 on 2021-08-22. Score: 1

From the phrasing, it seems that complicated refers to the non-convexity of the loss landscapes of neural networks. We do not have formal guarantees of convergence in general for such landscapes. This non-convexity is a property of both the function defined by the neural network, and the particular loss function we use.

In practice though, **non-convexity stems from the non-linear activation functions** as we almost exclusively use cross-entropy loss when training neural networks.


---

