Question: When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased.

I have read explanations related to the phenomenon, and it seems an increasing validation loss and validation accuracy signifies an overfitted model.

However, I have not really grokked the reasons why an increasing validation loss and validation accuracy signifies an overfitting.

Could you please give the explanations behind this phenomenon?

Answer: 
Consider a simple example where the cost function to be a parabola $y=x^2$ which is convex(ideal case) with a one global minima at $x=0$

Here your $y$ is the independent variable and $x$ is the dependent variable, analogus to the weights of model that you are trying to learn.

This is how it would look like.

[![enter image description here](https://i.stack.imgur.com/5zcMe.png)](https://i.stack.imgur.com/5zcMe.png)

Let's apply gradient descent to this particular cost function(parabola) to find it's minima.

From calculus it is clear that $dy/dx = 2\*x$. So that means that the gradients are positive in the $1^{st}$ quadrant and negative in the $2^{nd}$. That means for every **positive** small step in x that we take, we move away from origin in the $1^{st}$ quadrant and move towards the origin in the $2^{nd}$ quadrant(step is still positive).

In the update rule of gradient descent the '-' negative sign basically negates the gradient and hence always moves towards the local minima.

* $1^{st}$ quadrant -> gradient is positive, but if you use this as it is you move away from origin or minima. So, the negative sign helps here.
* $2^{nd}$ quadrant -> gradient is negative, but if you use this as it is you move away from origin or minima(addition of two negative values). So, the negative sign helps here.

Here is a small python code to make things clearer-

```py
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)
y = x**2

plt.xlabel('x')
plt.ylabel('y = x^2')
plt.plot(x, y)

# learning rate
lr = 0.1
np.random.seed(20)
x_start = np.random.normal(0, 2, 1)
dy_dx_old = 2 * x_start
dy_dx_new = 0

tolerance = 1e-2
# stop once the value has converged
while abs(dy_dx_new - dy_dx_old) > tolerance:
    dy_dx_old = dy_dx_new
    x_start = x_start - lr * dy_dx_old
    dy_dx_new = 2 * x_start
    plt.scatter(x_start, x_start**2)
    plt.pause(0.5)
plt.show()

```

[![enter image description here](https://i.stack.imgur.com/APPGu.png)](https://i.stack.imgur.com/APPGu.png)
Computing the gradient gives you the direction where the function increases the most. Consider f: x -> x^2 , the gradient in x=1 is 2, if you want to minimize the function you need to go in the direction of -2, same with x=-1 as the gradient is -2.

And as gradients are usually vectors, I don't know what a positive or negative gradient would be if gradient is something like (-1, 1).

<https://builtin.com/data-science/gradient-descent>