Answers.Body: <p>The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-03T10:00:13.830Z
Answers.LastActivityDate: 2020-08-03T10:00:13.830Z
Answers.UserDisplayName: SpiderRico
Body: <p>We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?</p>
  
  <p><a href="https://i.sstatic.net/eL6Hh.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eL6Hh.gif" alt="enter image description here"></a></p>
  
  <p>Here is a picture of what I mean by the Route.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T05:20:26.717Z
Comments.Text: Well, this is most likely application depended, what the developer expects from it etc. By route, I guess you mean the test/validation accuracy ?
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T05:49:33.187Z
Comments.Text: @SpiderRico Thank you for your response. I have added a picture to the question. Check that out for clarification about the route.
Comments.UserDisplayName: Ali Khalili
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T05:51:31.247Z
Comments.Text: Ah I see, you're talking of particular trajectories followed by algorithms on the loss surface. In this, I guess, what you're really interested is the generalization capacities of the minima found by these algorithms. Obviously, you'd want to pick the one that generalizes well/
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T06:32:04.173Z
Comments.Text: @SpiderRico Hmm, just to know we are on the same page, So the route can be a deciding factor too but it depends on the problem, right?
Comments.UserDisplayName: Ali Khalili
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T06:35:52.713Z
Comments.Text: The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of accuracy performance, is the final point which the trajectory converges.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T07:22:14.250Z
Comments.Text: @SpiderRico Thanks for the response. Fully understood.
Comments.UserDisplayName: Ali Khalili
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-31T07:23:48.663Z
Comments.Text: Sure thing. Happy to help.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T13:30:53.770Z
Comments.Text: @SpiderRico I think you have answered the question here. Could you please provide and write an answer?
Comments.UserDisplayName: pedrum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T13:34:09.090Z
Comments.Text: @AliKHalili nice pic. Just one point I wanted to mention here is that you are getting the same result in the end in your example. Can you think of any example where e.g. blue rout could problematic?
Comments.UserDisplayName: pedrum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-29T16:46:29.170Z
Comments.Text: @pedrum Unfortunately, I can't give you an example but I can say there could be cases where, if you use different loss optimization you won't get the same result. If we look at the case a little closer, the reason for this can be attributed to data problems. For example, if you don't pre-process your data properly you might get different results at the end. Or even in some cases as mentioned above, your data might be very sensitive to the optimization you use, and you might get different results, or should I say bad result if you use blue line, even if you pre-process your data correctly.
Comments.UserDisplayName: Ali Khalili
Tags: <convolutional-neural-networks><optimization>
Title: Which one is more important in case of different loss optimization algorithms, Speed or the Route?
UserDisplayName: Ali Khalili
ThreadText: # Which one is more important in case of different loss optimization algorithms, Speed or the Route?
  
  (Asked by: username_0 on 2020-05-28)
  
  We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?
  
  [![enter image description here](https://i.sstatic.net/eL6Hh.gif)](https://i.sstatic.net/eL6Hh.gif)
  
  Here is a picture of what I mean by the Route.
  
  **Tags:** convolutional-neural-networks, optimization
  > **Comments:**
  > **username_1**: Well, this is most likely application depended, what the developer expects from it etc. By route, I guess you mean the test/validation accuracy ?  
  > **username_0**: @SpiderRico Thank you for your response. I have added a picture to the question. Check that out for clarification about the route.  
  > **username_1**: Ah I see, you're talking of particular trajectories followed by algorithms on the loss surface. In this, I guess, what you're really interested is the generalization capacities of the minima found by these algorithms. Obviously, you'd want to pick the one that generalizes well/  
  > **username_0**: @SpiderRico Hmm, just to know we are on the same page, So the route can be a deciding factor too but it depends on the problem, right?  
  > **username_1**: The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of accuracy performance, is the final point which the trajectory converges.  
  > **username_0**: @SpiderRico Thanks for the response. Fully understood.  
  > **username_1**: Sure thing. Happy to help.  
  > **username_2**: @SpiderRico I think you have answered the question here. Could you please provide and write an answer?  
  > **username_2**: @AliKHalili nice pic. Just one point I wanted to mention here is that you are getting the same result in the end in your example. Can you think of any example where e.g. blue rout could problematic?  
  > **username_0**: @pedrum Unfortunately, I can't give you an example but I can say there could be cases where, if you use different loss optimization you won't get the same result. If we look at the case a little closer, the reason for this can be attributed to data problems. For example, if you don't pre-process your data properly you might get different results at the end. Or even in some cases as mentioned above, your data might be very sensitive to the optimization you use, and you might get different results, or should I say bad result if you use blue line, even if you pre-process your data correctly.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-03. Score: 1 | Accepted answer
  
  The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.
  
  
  ---
  
  