Answers.Body: <p>It seems like the model is falling into a bad local minimum, where the model is stuck in an easy-to-learn &quot;good-enough&quot; solution. Increasing the initial learning rate could fix this.</p>
  <p>See <a href="https://ai.stackexchange.com/questions/10013/what-is-a-bad-local-minimum-in-machine-learning">this post</a> for more information on this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T14:08:44.993Z
Answers.LastActivityDate: 2023-08-03T14:08:44.993Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I'm currently training a CNN + multiple target regression model that does the following</p>
  <p>input: <span class="math-container">$ \dim x = (L, 2), \text{where} \ x_i \in (-0.1, 0.1) $</span>  <br />
  output: <span class="math-container">$\dim y = (M), \text{where} \ y_i \geq 0, y_{i-1} \leq y_i$</span> <br />
  with <span class="math-container">$10&lt; L, M &lt; 100$</span></p>
  <p>The basic architecture is (each step with ReLU activation) : <br />
  input <span class="math-container">$\rightarrow$</span> CNN1(3x2) <span class="math-container">$\rightarrow$</span> maxpool(2x2) <span class="math-container">$\rightarrow$</span> CNN2(3x2) <span class="math-container">$\rightarrow$</span> maxpool(2x2) <span class="math-container">$\rightarrow$</span> flatten <span class="math-container">$\rightarrow$</span> hidden1 <span class="math-container">$\rightarrow$</span> hidden2 <span class="math-container">$\rightarrow$</span> M outputs</p>
  <p>The issue is that the model is completely insensitive to the variations in the input data, i.e., regardless of the input value, the model would predict an output that is almost the average of all training outputs.</p>
  <p>Some hyperparameters:</p>
  <pre><code>batch = 20
  epoch = 5
  lr = 0.05
  decay = 0.02
  momentum = 0.9
  </code></pre>
  <p>The training errors are not really lowering after the first few epochs, so I think it could be stuck in a local minima.</p>
  <p>Here I'm plotting 10 random predictions on the test set data
  <a href="https://i.sstatic.net/3YMKu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3YMKu.png" alt="Plotting 10 random predicts on the test set data" /></a></p>
  <p>Looks to me that there is not enough distinction in the inputs for the model to recognize, or that the input simply looks like noise to the model. I've tried some rudimentary data augmentation techniques, exponentiating the differences, normalization, etc. to no avail.</p>
  <p>The training data is generated by diagonalizing matrices parametrized by input values, and the lowest <span class="math-container">$M$</span> eigenvalues are the output values.</p>
  <p>Since my background is not in the field, I would appreciate any advice and suggestions. Thanks in advance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-06T10:30:55.590Z
Comments.Text: for how many epochs are you training it? with which optimizer/learning rate? not clear what you are doing, but are you sure that there is enough information in the input to predict the output?
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-08T04:06:06.707Z
Comments.Text: @AlbertoSinigaglia Please see the parameters above. As the answer below suggested, a larger learning rate sometimes results in weights becoming NaN, presumably some gradient explosion (?)
  
  The inputs generates a Hermitian matrix (they alone do not generate the matrix, but the additional information needed for such generation is identical across all cases), and I'm simulating the process of solving (diagonalizing) that matrix, but that's a good question. I will look into that more closely
Comments.UserDisplayName: RLLL
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-04T21:10:57.063Z
Comments.Text: What loss function did you use? This could be *the* key here
Comments.UserDisplayName: Ggjj11
Tags: <machine-learning><convolutional-neural-networks><regression>
Title: CNN-Regression insensitive to input data
UserDisplayName: RLLL
ThreadText: # CNN-Regression insensitive to input data
  
  (Asked by: username_0 on 2023-08-03)
  
  I'm currently training a CNN + multiple target regression model that does the following
  
  input: $ \dim x = (L, 2), \text{where} \ x\_i \in (-0.1, 0.1) $   
  output: $\dim y = (M), \text{where} \ y\_i \geq 0, y\_{i-1} \leq y\_i$   
  with $10< L, M < 100$
  
  The basic architecture is (each step with ReLU activation) :   
  input $\rightarrow$ CNN1(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ CNN2(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ flatten $\rightarrow$ hidden1 $\rightarrow$ hidden2 $\rightarrow$ M outputs
  
  The issue is that the model is completely insensitive to the variations in the input data, i.e., regardless of the input value, the model would predict an output that is almost the average of all training outputs.
  
  Some hyperparameters:
  
  ```
  batch = 20
  epoch = 5
  lr = 0.05
  decay = 0.02
  momentum = 0.9
  
  ```
  
  The training errors are not really lowering after the first few epochs, so I think it could be stuck in a local minima.
  
  Here I'm plotting 10 random predictions on the test set data
  [![Plotting 10 random predicts on the test set data](https://i.sstatic.net/3YMKu.png)](https://i.sstatic.net/3YMKu.png)
  
  Looks to me that there is not enough distinction in the inputs for the model to recognize, or that the input simply looks like noise to the model. I've tried some rudimentary data augmentation techniques, exponentiating the differences, normalization, etc. to no avail.
  
  The training data is generated by diagonalizing matrices parametrized by input values, and the lowest $M$ eigenvalues are the output values.
  
  Since my background is not in the field, I would appreciate any advice and suggestions. Thanks in advance.
  
  **Tags:** machine-learning, convolutional-neural-networks, regression
  
  > **Comments:**
  > **username_1**: for how many epochs are you training it? with which optimizer/learning rate? not clear what you are doing, but are you sure that there is enough information in the input to predict the output?  
  > **username_0**: @AlbertoSinigaglia Please see the parameters above. As the answer below suggested, a larger learning rate sometimes results in weights becoming NaN, presumably some gradient explosion (?)  The inputs generates a Hermitian matrix (they alone do not generate the matrix, but the additional information needed for such generation is identical across all cases), and I'm simulating the process of solving (diagonalizing) that matrix, but that's a good question. I will look into that more closely  
  > **username_2**: What loss function did you use? This could be *the* key here  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-08-03. Score: 0
  
  It seems like the model is falling into a bad local minimum, where the model is stuck in an easy-to-learn "good-enough" solution. Increasing the initial learning rate could fix this.
  
  See [this post](https://ai.stackexchange.com/questions/10013/what-is-a-bad-local-minimum-in-machine-learning) for more information on this.
  
  
  ---
  
  