<p>The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.</p>

CC BY-SA 4.0
2020-08-03T10:00:13.830Z
2020-08-03T10:00:13.830Z
SpiderRico
<p>We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?</p>

<p><a href="https://i.sstatic.net/eL6Hh.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eL6Hh.gif" alt="enter image description here"></a></p>

<p>Here is a picture of what I mean by the Route.</p>

CC BY-SA 4.0
2020-05-31T05:20:26.717Z
Well, this is most likely application depended, what the developer expects from it etc. By route, I guess you mean the test/validation accuracy ?
SpiderRico
CC BY-SA 4.0
2020-05-31T05:49:33.187Z
@SpiderRico Thank you for your response. I have added a picture to the question. Check that out for clarification about the route.
Ali Khalili
CC BY-SA 4.0
2020-05-31T05:51:31.247Z
Ah I see, you're talking of particular trajectories followed by algorithms on the loss surface. In this, I guess, what you're really interested is the generalization capacities of the minima found by these algorithms. Obviously, you'd want to pick the one that generalizes well/
SpiderRico
CC BY-SA 4.0
2020-05-31T06:32:04.173Z
@SpiderRico Hmm, just to know we are on the same page, So the route can be a deciding factor too but it depends on the problem, right?
Ali Khalili
CC BY-SA 4.0
2020-05-31T06:35:52.713Z
The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of accuracy performance, is the final point which the trajectory converges.
SpiderRico
CC BY-SA 4.0
2020-05-31T07:22:14.250Z
@SpiderRico Thanks for the response. Fully understood.
Ali Khalili
CC BY-SA 4.0
2020-05-31T07:23:48.663Z
Sure thing. Happy to help.
SpiderRico
CC BY-SA 4.0
2020-07-28T13:30:53.770Z
@SpiderRico I think you have answered the question here. Could you please provide and write an answer?
pedrum
CC BY-SA 4.0
2020-07-28T13:34:09.090Z
@AliKHalili nice pic. Just one point I wanted to mention here is that you are getting the same result in the end in your example. Can you think of any example where e.g. blue rout could problematic?
pedrum
CC BY-SA 4.0
2020-07-29T16:46:29.170Z
@pedrum Unfortunately, I can't give you an example but I can say there could be cases where, if you use different loss optimization you won't get the same result. If we look at the case a little closer, the reason for this can be attributed to data problems. For example, if you don't pre-process your data properly you might get different results at the end. Or even in some cases as mentioned above, your data might be very sensitive to the optimization you use, and you might get different results, or should I say bad result if you use blue line, even if you pre-process your data correctly.
Ali Khalili
<convolutional-neural-networks><optimization>
Which one is more important in case of different loss optimization algorithms, Speed or the Route?
Ali Khalili
# Which one is more important in case of different loss optimization algorithms, Speed or the Route?

(Asked by: username_0 on 2020-05-28)

We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?

[![enter image description here](https://i.sstatic.net/eL6Hh.gif)](https://i.sstatic.net/eL6Hh.gif)

Here is a picture of what I mean by the Route.

**Tags:** convolutional-neural-networks, optimization

> **Comments:**
> **username_1**: Well, this is most likely application depended, what the developer expects from it etc. By route, I guess you mean the test/validation accuracy ?  
> **username_0**: @username_1 Thank you for your response. I have added a picture to the question. Check that out for clarification about the route.  
> **username_1**: Ah I see, you're talking of particular trajectories followed by algorithms on the loss surface. In this, I guess, what you're really interested is the generalization capacities of the minima found by these algorithms. Obviously, you'd want to pick the one that generalizes well/  
> **username_0**: @username_1 Hmm, just to know we are on the same page, So the route can be a deciding factor too but it depends on the problem, right?  
> **username_1**: The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of accuracy performance, is the final point which the trajectory converges.  
> **username_0**: @username_1 Thanks for the response. Fully understood.  
> **username_1**: Sure thing. Happy to help.  
> **username_2**: @username_1 I think you have answered the question here. Could you please provide and write an answer?  
> **username_2**: @AliKHalili nice pic. Just one point I wanted to mention here is that you are getting the same result in the end in your example. Can you think of any example where e.g. blue rout could problematic?  
> **username_0**: @username_2 Unfortunately, I can't give you an example but I can say there could be cases where, if you use different loss optimization you won't get the same result. If we look at the case a little closer, the reason for this can be attributed to data problems. For example, if you don't pre-process your data properly you might get different results at the end. Or even in some cases as mentioned above, your data might be very sensitive to the optimization you use, and you might get different results, or should I say bad result if you use blue line, even if you pre-process your data correctly.  

---

## 1 answers

### Answer by username_1 on 2020-08-03. Score: 1 | Accepted answer

The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.


---

