Answers.Body: <p>How about:</p>
  <ul>
  <li>speed of convergence</li>
  <li>stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate)</li>
  <li>presence/number of saddle points in your loss landscape</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T13:29:57.713Z
Answers.LastActivityDate: 2022-06-16T13:29:57.713Z
Answers.UserDisplayName: ImotVoksim
Body: <p>I'm comparing 2 optimization algorithms for deep neural nets through visualizing the loss landscape. The visualization method is described <a href="https://arxiv.org/abs/1712.09913" rel="nofollow noreferrer">here</a>.</p>
  <p>Besides the qualitative observation that how trajectory moves w.r.t. the loss level-sets, are there any quantitive measures to compare the two methods?</p>
  <p><a href="https://i.sstatic.net/28MPDm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/28MPDm.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/4rbdzm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4rbdzm.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><loss>
Title: Visualizing the loss landscape in deep NN to compare optimization methods
UserDisplayName: Blade
ThreadText: # Visualizing the loss landscape in deep NN to compare optimization methods
  
  (Asked by: username_0 on 2022-06-14)
  
  I'm comparing 2 optimization algorithms for deep neural nets through visualizing the loss landscape. The visualization method is described [here](https://arxiv.org/abs/1712.09913).
  
  Besides the qualitative observation that how trajectory moves w.r.t. the loss level-sets, are there any quantitive measures to compare the two methods?
  
  [![enter image description here](https://i.sstatic.net/28MPDm.png)](https://i.sstatic.net/28MPDm.png)
  [![enter image description here](https://i.sstatic.net/4rbdzm.png)](https://i.sstatic.net/4rbdzm.png)
  
  **Tags:** neural-networks, loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-16. Score: 1
  
  How about:
  
  * speed of convergence
  * stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate)
  * presence/number of saddle points in your loss landscape
  
  
  ---
  
  