Why use MSE instead of SSE as cost function in linear regression? I am studying linear regression and I solved some problems analytically.
For that I used the normal and intuitive sum of squared error function.
Looking at this function, it makes all sense why it looks the way it does. I mean we have the square because we can have positive and negative errors and we do not want that they eliminate each other and we use the square, because we get exactly one minima, which is a solvable minimization problem.
We can use also gradient descent.
Looking at other cost functions, like the mean squared error, I do not understand why it is a good idea to calculate the mean of errors, I mean what motivates us to do this?
I mean it is just an optimization problem, we want the best fit, so I would never get the idea to use the mean of errors and why it makes our optimization problem better.
Is it just to work with a smaller error values, so when using gradient descent it converts faster?
would be great if someone could help me with the intuition and the mathematical motivation.
thanks in advance!

Let the mean squared-error (MSE) cost function be
$$
\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^N (y_i - f(x_i,\theta))^2
$$
where $x_i$ represents the $i^{\text{th}}$ input, $y_i$ represents the $i^{\text{th}}$ target, and $\theta$ represents the parameters. The goal is to find the $\theta$ that minimizes the MSE cost function.
The number $\frac{1}{N}$ does not change which $\theta$ minimizes the MSE cost function, since it just scales the cost. However, as you have guessed already, it plays a practical role when performing gradient descent, or any other iterative procedure involving gradients and/or Hessians.
The gradient descent update rule is
$$
\theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \mathcal{L}(\theta)
$$
where $\nabla_{\theta} \mathcal{L}(\theta)$ is the gradient of the MSE cost function with respect to $\theta$. In any descent procedure, you have to choose two things: the direction that you want to descend, and how much you want to descend in that direction. In the gradient descent update rule above, we have already chosen the direction, it is $\nabla_{\theta} \mathcal{L}(\theta)$. However, we have not yet chosen how much we want to descend in that direction. This is done by choosing an appropriate $\alpha$, also known as the "step size".
How do we pick an appropriate $\alpha$? This is what line search is for. In line search, specifically backtracking line search, we want to choose the $\alpha$ that satisfies some specific conditions. It turns out that if the gradient $\nabla_{\theta} \mathcal{L}(\theta)$ is large, such as when you are far away from the origin on a parabola, it takes more time/computation to find an $\alpha$ value that satisfies these conditions. Therefore, we ideally want the values of $\nabla_{\theta} \mathcal{L}(\theta)$ to be small.
The MSE cost function inherently keeps $\nabla_{\theta} \mathcal{L}(\theta)$ small using $\frac{1}{N}$. To see this, suppose that we instead use the sum of squared-errors (SSE) cost function
$$
\tilde{\mathcal{L}}(\theta) = \sum_{i=1}^N (y_i - f(x_i,\theta))^2
$$
and so the gradient descent update rule becomes
$$
\theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \tilde{\mathcal{L}}(\theta)
$$
To keep $\nabla_{\theta} \tilde{\mathcal{L}}(\theta)$ small, let us multiply it by $\frac{1}{N}$ such that
$$
\theta_{k+1} \leftarrow \theta_k - \alpha \cdot \frac{1}{N}\nabla_{\theta} \tilde{\mathcal{L}}(\theta)
$$
Since the gradient operator is linear, this becomes
$$
\theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \frac{1}{N} \tilde{\mathcal{L}}(\theta)
$$
But since
$$
\frac{1}{N} \tilde{\mathcal{L}}(\theta) = \mathcal{L}(\theta)
$$
In other words, $\frac{1}{N}$ times the SSE cost function is the MSE cost function. So, the gradient descent update rule with small gradients is
$$
\theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta}\mathcal{L}(\theta)
$$
In fact, notice that as $N$ grows, the SSE cost function grows too, and so does its gradient. Conversely, while the MSE cost function may also grow in value as $N$ increases, it does not grow as fast as the SSE cost function, and so its gradients are also growing slowly.
