Answers.Body: <p>It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.</p>
  
  <p>And also, the 3D space can't be used to plot more than 3 dimensions.</p>
  
  <p>However, with a deep network with lots of weights, these can be plotted:</p>
  
  <ul>
  <li>Loss value against any pair of 2 weights</li>
  <li>Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)</li>
  </ul>
  
  <p>Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):</p>
  
  <p><a href="https://i.sstatic.net/0XeMB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0XeMB.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-18T07:59:15.020Z
Answers.LastActivityDate: 2019-10-18T07:59:15.020Z
Answers.UserDisplayName: Dan D.
Body: <p>For a single neuron with <strong>2 weights</strong>, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):</p>
  
  <p><a href="https://i.sstatic.net/vG2Vp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vG2Vp.png" alt="enter image description here"></a></p>
  
  <p>But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-18T04:17:42.417Z
Comments.Text: I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-18T04:30:10.943Z
Comments.Text: @Recessive oh, but i can see some very complex loss landscape this way: https://www.pyimagesearch.com/wp-content/uploads/2019/10/train_val_loss_landscape.png
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-18T04:31:36.427Z
Comments.Text: the image in the comment right above is from this article: https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-18T06:44:57.310Z
Comments.Text: Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: http://firsttimeprogrammer.blogspot.com/search/label/Machine%20Learning. You'll see this is a graph for `f(x,y) = x**2 + y**2 –2*x*y`. You can see the (falsely) modified image halfway down this page: https://medium.com/@RosieCampbell/demystifying-deep-neural-nets-efb726eae941. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation.
Comments.UserDisplayName: Recessive
Tags: <machine-learning><gradient-descent><plotting><loss><weights>
Title: How to plot Loss Landscape with more than 2 weights in the network
UserDisplayName: Dan D.
ThreadText: # How to plot Loss Landscape with more than 2 weights in the network
  
  (Asked by: username_0 on 2019-10-18)
  
  For a single neuron with **2 weights**, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):
  
  [![enter image description here](https://i.sstatic.net/vG2Vp.png)](https://i.sstatic.net/vG2Vp.png)
  
  But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?
  
  **Tags:** machine-learning, gradient-descent, plotting, loss, weights
  > **Comments:**
  > **username_1**: I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more  
  > **username_0**: @Recessive oh, but i can see some very complex loss landscape this way: https://www.pyimagesearch.com/wp-content/uploads/2019/10/train_val_loss_landscape.png  
  > **username_0**: the image in the comment right above is from this article: https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/  
  > **username_1**: Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: http://firsttimeprogrammer.blogspot.com/search/label/Machine%20Learning. You'll see this is a graph for `f(x,y) = x**2 + y**2 –2*x*y`. You can see the (falsely) modified image halfway down this page: https://medium.com/@RosieCampbell/demystifying-deep-neural-nets-efb726eae941. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-10-18. Score: 0 | Accepted answer
  
  It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.
  
  And also, the 3D space can't be used to plot more than 3 dimensions.
  
  However, with a deep network with lots of weights, these can be plotted:
  
  * Loss value against any pair of 2 weights
  * Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)
  
  Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):
  
  [![enter image description here](https://i.sstatic.net/0XeMB.png)](https://i.sstatic.net/0XeMB.png)
  
  
  ---
  
  