Answers.Body: <p>I think the primary solution to plateauing is improving the dataset.</p>
  <p>The iteration process should be</p>
  <ol>
  <li>Train the neural network</li>
  <li>Identify what properties are causing the neural network to plateau.</li>
  <li>Gather more data, improve the targets, argument the existing dataset, to target these undesired properties.</li>
  <li>Find the next plateau, and repeat until the desired results are achieved.</li>
  </ol>
  <p>We are changing the <a href="https://youtu.be/78vq6kgsTa8?t=978" rel="nofollow noreferrer">loss landscape</a> using the dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T20:33:18.347Z
Answers.LastActivityDate: 2022-11-03T20:33:18.347Z
Answers.UserDisplayName: Tom Huntington
Body: <p>Below is the loss of the same training run at different scales illustrating the plateau phenomenon.
  <img src="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7759d9f-88f2-4a96-a38a-82e281b7415e_1024x300.png" alt="" />
  <a href="https://arxiv.org/pdf/2007.07213.pdf" rel="nofollow noreferrer">Source p3</a>.</p>
  <p>It seems to me that each dip adds constraints to the neural network optimization, and that neural networks must be highly over parameterized to enable the optimizer to find paths through to the next dip.</p>
  <p>I am trying to train timeseries features for a covariance matrix (i.e. <code>einsum('bic,bjc-&gt;bij', model(s0), model(s1))</code>). The first dip corresponds to the neural net maximizing the cosine similarities, while being linear in the magnitude i.e. <span class="math-container">$\gamma\lVert s \rVert =\lVert model(s)\rVert$</span>. The next dip corresponds to the the neural network being nonlinear in magnitude (i.e. silence, and everything else, being mapped onto the unit sphere).</p>
  <p>Here are a few responses to plateauing</p>
  <ol>
  <li>Regularize the architecture (an architecture that is non-linear in magnitude)</li>
  <li>Regularize the loss function (punish being linear in magnitude)</li>
  <li>Iterate on the training set to find architectures/hyperparameters that plateau less.</li>
  <li>Brute force it. Just leave the neural network training.</li>
  </ol>
  <p>I'm sceptical about 1 and 2 as it would only get you past the first plateau. I don't hear much about 3, although I think it might be a good idea. Does anyone do this in practice?</p>
  
Tags: <neural-networks>
Title: Alternatives to brute forcing neural network plateau
UserDisplayName: Tom Huntington
ThreadText: # Alternatives to brute forcing neural network plateau
  
  (Asked by: username_0 on 2022-11-02)
  
  Below is the loss of the same training run at different scales illustrating the plateau phenomenon.
  ![](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7759d9f-88f2-4a96-a38a-82e281b7415e_1024x300.png)
  [Source p3](https://arxiv.org/pdf/2007.07213.pdf).
  
  It seems to me that each dip adds constraints to the neural network optimization, and that neural networks must be highly over parameterized to enable the optimizer to find paths through to the next dip.
  
  I am trying to train timeseries features for a covariance matrix (i.e. `einsum('bic,bjc->bij', model(s0), model(s1))`). The first dip corresponds to the neural net maximizing the cosine similarities, while being linear in the magnitude i.e. $\gamma\lVert s \rVert =\lVert model(s)\rVert$. The next dip corresponds to the the neural network being nonlinear in magnitude (i.e. silence, and everything else, being mapped onto the unit sphere).
  
  Here are a few responses to plateauing
  
  1. Regularize the architecture (an architecture that is non-linear in magnitude)
  2. Regularize the loss function (punish being linear in magnitude)
  3. Iterate on the training set to find architectures/hyperparameters that plateau less.
  4. Brute force it. Just leave the neural network training.
  
  I'm sceptical about 1 and 2 as it would only get you past the first plateau. I don't hear much about 3, although I think it might be a good idea. Does anyone do this in practice?
  
  **Tags:** neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-11-03. Score: -1
  
  I think the primary solution to plateauing is improving the dataset.
  
  The iteration process should be
  
  1. Train the neural network
  2. Identify what properties are causing the neural network to plateau.
  3. Gather more data, improve the targets, argument the existing dataset, to target these undesired properties.
  4. Find the next plateau, and repeat until the desired results are achieved.
  
  We are changing the [loss landscape](https://youtu.be/78vq6kgsTa8?t=978) using the dataset.
  
  
  ---
  
  