Body: <p>I am training a standard feed-forward, fully connect neural network on ~30,000 6d input vectors paired to 30,000 5d output vectors. The vectors capture position and speed of a classical mechanics system.</p>
  <p>I am using PyTorch, and after much trial and error, have found the following neural network seems to work OK:</p>
  <pre><code>self.neural_net = nn.Sequential(
              nn.Linear(6,1000), nn.ReLU(), 
  
              nn.Linear(1000, 1000), nn.Sigmoid(),
              nn.Linear(1000, 1000), nn.Sigmoid(),
          
              nn.Linear(1000, 5)
          )
  </code></pre>
  <p>Work &quot;OK&quot; means: the loss goes down pretty low, using a MSE loss function, batch size=300, lr=1e-4, SGD optimizer. See the top plot here:</p>
  <p><a href="https://i.sstatic.net/2fpu4kWM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2fpu4kWM.png" alt="Loss profile of training a neural network" /></a></p>
  <p>During training, I feed the network a random selection of 400 of the 30,000 input vectors. As it trains the network's ability to recognize the 400 vectors fluctuates very wildly, as shown in the bottom plot.</p>
  <p>Does anyone know why the fluctuations are so extreme? Does this have something to do with my network's structure? I have tried messing around with regularization and different activations, etc. to no avail.</p>
  <p>I see the fluctuation envelope decreasing as the loss goes down, and it seems like if I just wait another 10,000 training epochs, I'll be good, so maybe it's just the gradient descent doing its job?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-19T21:20:46.653Z
Comments.Text: Try to lower learning rate and batch_size (64 or 128) and hidden layers which tend to cause vanishing gradient for sigmoid activation. Try ReLU or Leaky ReLU, and BatchNorm with Early Stopping.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-20T00:23:13.270Z
Comments.Text: Thanks very much. I'll give your suggestions a try. (At the moment, using [Leaky]ReLU throughout causes a loss=nan to occur.)
Comments.UserDisplayName: Tom Bensky
Tags: <neural-networks><machine-learning><training><loss>
Title: Why are the fluctuations in this plot so extreme?
UserDisplayName: Tom Bensky
ThreadText: # Why are the fluctuations in this plot so extreme?
  
  (Asked by: username_0 on 2024-12-19)
  
  I am training a standard feed-forward, fully connect neural network on ~30,000 6d input vectors paired to 30,000 5d output vectors. The vectors capture position and speed of a classical mechanics system.
  
  I am using PyTorch, and after much trial and error, have found the following neural network seems to work OK:
  
  ```
  self.neural_net = nn.Sequential(
              nn.Linear(6,1000), nn.ReLU(), 
  
              nn.Linear(1000, 1000), nn.Sigmoid(),
              nn.Linear(1000, 1000), nn.Sigmoid(),
          
              nn.Linear(1000, 5)
          )
  
  ```
  
  Work "OK" means: the loss goes down pretty low, using a MSE loss function, batch size=300, lr=1e-4, SGD optimizer. See the top plot here:
  
  [![Loss profile of training a neural network](https://i.sstatic.net/2fpu4kWM.png)](https://i.sstatic.net/2fpu4kWM.png)
  
  During training, I feed the network a random selection of 400 of the 30,000 input vectors. As it trains the network's ability to recognize the 400 vectors fluctuates very wildly, as shown in the bottom plot.
  
  Does anyone know why the fluctuations are so extreme? Does this have something to do with my network's structure? I have tried messing around with regularization and different activations, etc. to no avail.
  
  I see the fluctuation envelope decreasing as the loss goes down, and it seems like if I just wait another 10,000 training epochs, I'll be good, so maybe it's just the gradient descent doing its job?
  
  **Tags:** neural-networks, machine-learning, training, loss
  
  > **Comments:**
  > **username_1**: Try to lower learning rate and batch_size (64 or 128) and hidden layers which tend to cause vanishing gradient for sigmoid activation. Try ReLU or Leaky ReLU, and BatchNorm with Early Stopping.  
  > **username_0**: Thanks very much. I'll give your suggestions a try. (At the moment, using [Leaky]ReLU throughout causes a loss=nan to occur.)  
  
  ---
  
  ## 0 answers
  
  