Body: <p>I have two convex, smooth loss functions to minimise. During the training (a very simple model) using batch SGD (with tuned optimal learning rate for each loss function), I observe that the (log) loss curve of the loss 2 converges much faster and is much more smooth than that of the loss 2, as shown in the figure.</p>
  
  <p>What can I say more about the properties of the two loss functions, for example in terms of smoothness, convexity, etc?</p>
  
  <p><a href="https://i.sstatic.net/vjD9R.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vjD9R.jpg" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-05T00:59:17.547Z
Comments.Text: I'll ask the same question I asked in the data science SE : what make you think you can get overall properties of functions with a set of evaluations along an optimisation path ? From just your graph it's not possible to tell that much, yeah the minima of loss 1 appears lower than the one of loss 2. The path appears bumpier but it may be due on how you selected your 'optimal' learning rates. Even with a lot more details we probably won't be able to answer that question.
Comments.UserDisplayName: Lucas Morin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-05T01:16:46.893Z
Comments.Text: The loss 1 is in fact smaller than the loss 2, by construction. As I said, the learning rate seems to impact only the speed of convergence, not the smoothness of the loss curve. I've also tried other datasets and observed the same behaviors. I know that it's not enough to generalise from what I observe, but I want to understand why such behaviors persist in the datasets I'm working with.
Comments.UserDisplayName: SiXUlm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-05T01:24:02.937Z
Comments.Text: Maybe start by giving us the form of your loss and be more precise about what you want to know (that would be not trivially given by the loss forms).
Comments.UserDisplayName: Lucas Morin
Tags: <training><objective-functions><loss>
Title: Deduce properties of the loss functions from the training loss curves
UserDisplayName: SiXUlm
ThreadText: # Deduce properties of the loss functions from the training loss curves
  
  (Asked by: username_0 on 2020-01-05)
  
  I have two convex, smooth loss functions to minimise. During the training (a very simple model) using batch SGD (with tuned optimal learning rate for each loss function), I observe that the (log) loss curve of the loss 2 converges much faster and is much more smooth than that of the loss 2, as shown in the figure.
  
  What can I say more about the properties of the two loss functions, for example in terms of smoothness, convexity, etc?
  
  [![enter image description here](https://i.sstatic.net/vjD9R.jpg)](https://i.sstatic.net/vjD9R.jpg)
  
  **Tags:** training, objective-functions, loss
  > **Comments:**
  > **username_1**: I'll ask the same question I asked in the data science SE : what make you think you can get overall properties of functions with a set of evaluations along an optimisation path ? From just your graph it's not possible to tell that much, yeah the minima of loss 1 appears lower than the one of loss 2. The path appears bumpier but it may be due on how you selected your 'optimal' learning rates. Even with a lot more details we probably won't be able to answer that question.  
  > **username_0**: The loss 1 is in fact smaller than the loss 2, by construction. As I said, the learning rate seems to impact only the speed of convergence, not the smoothness of the loss curve. I've also tried other datasets and observed the same behaviors. I know that it's not enough to generalise from what I observe, but I want to understand why such behaviors persist in the datasets I'm working with.  
  > **username_1**: Maybe start by giving us the form of your loss and be more precise about what you want to know (that would be not trivially given by the loss forms).  
  
  ---
  
  ## 0 answers
  
  