Answers.Body: <p>The core issue here is that the two parameters you want to predict have actually different difficulty levels for a computer vision system. You have created a neural network model capable of solving the easier one, but not the more difficult one.</p>
  <p>The scale factor is relatively easy to estimate because it (inversely) correlates closely to the number of simple features in the image. A simple flexible corner detection filter, provided the scale is not too low, will when flattened produce an almost linear relationship based on counting the features. A very basic CNN, like the one in the question, should be capable of making this kind of calculation.</p>
  <p>The randomness factor is harder - it is going to be correlated in a more complex way to the <em>variance</em> of the corner and line features in the image. A variance is harder to calculate than a count of matches, so will likely require a more complex network, and more training examples.</p>
  <p>In addition, from comments, the scaling of the two parameters is radically different, which will cause a problem with processing the error signal. When predicting both values at once you should probably normalise the outputs so that one error signal does not dominate the neural network learning.</p>
  <p>Where to go from here? Here are a few suggestions:</p>
  <ul>
  <li>Isolate the problem by only predicting the randomness variable. Then you can make the joint prediction of both paramaters as a later piece of work, once you have a better idea what works separately.</li>
  <li>You need to perform a hyperparameter search - trying different neural network structures to see which perform better at the task. Initially I would recommend making a deeper fully-connected section (i.e. add a couple more <code>Dense(128,...</code> layers after the first one , but you can also play with the number of convolutional layers, and the number of features in each layer. More is not necessarily better, so you can try increasing number of layers, but decreasing the number of features or the number of virtual neurons.
  <ul>
  <li>Right now, whilst you are still learning, you can perform this search manually by editing your script, keeping notes on changes that seem to improve performance.</li>
  </ul>
  </li>
  <li>I would be concerned that 1000 training examples is not many for an image prediction task trained from scratch. Multiply by at least 10. Then you can check whether adding more examples improves the model by plotting a learning curve - train with 1000, 2000, 3000, 4000 etc, and see whether results improve. If results are still improving when moving from 9000 to 10000 training examples, it's worth going to 20000 total.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T18:33:58.890Z
Answers.LastActivityDate: 2023-08-24T18:33:58.890Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Stacking ReLUs will cause &quot;explosive&quot; gradients, since their gains get multiplied for the outputs which are on the positive range. So I suggest that you add BatchNormalization layers there in-between. And because the dimension after the <code>Flatten</code> is 246k, the next Dense layer ends up having 31.5 million parameters! Using global average and global max pooling, I got a model with just 40k parameters to fit the data very nicely. But sure, our data generation codes aren't identical. Here are some sample images:</p>
  <p><a href="https://i.sstatic.net/BEYmp.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BEYmp.jpg" alt="vornoi with noise" /></a></p>
  <p>Here are scatter plots on predicted (x-axis) and correct (y-axis, with added noise) values. RMSE values are 0.08 and 0.17, so indeed the noise level was a bit more difficult to estimate. In the training data I scaled both parameters to the scale from 0 to 1.</p>
  <p><a href="https://i.sstatic.net/YVH2x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YVH2x.png" alt="result scatter plots" /></a></p>
  <p>But the correlation-coefficient is so close to zero, that I started wondering whether you have mixed up your training data, and the target value was essentially random.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T15:00:06.950Z
Answers.Text: that looks interesting - would you mind sharing some code? I can provide the code for training data if that's beneficial
Answers.UserDisplayName: simone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T15:47:39.067Z
Answers.Text: Here is my script: https://gist.github.com/nikonyrh/df50f1eb2490af7df685fa263ae5ad36 . If you are still having problems, I could try using your script to generate the data and see what is the accuracy.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T17:52:40.553Z
Answers.Text: Awesome. I don't think I''l be able to test this before end of next week but I'll sure let you know
Answers.UserDisplayName: simone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T14:46:51.307Z
Answers.LastActivityDate: 2023-09-10T14:46:51.307Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I am trying to use Keras to estimate parameters used in the generation of Voronoi textures. I got good results in estimating the scale parameter, but extremely poor ones in the estimation of the randomness parameter (a correlation of 0.0551 and an RMSE of 0.5088 for a variable that ranges from 0 to 1).</p>
  <p>This is the model I am using:</p>
  <pre><code>model = Sequential([
      Conv2D(32, (3, 3), activation='relu', input_shape=(256, 256, 1)),
      MaxPooling2D((2, 2)),
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D((2, 2)),
      Flatten(),
      Dense(128, activation='relu'),
      Dense(var_count, activation='linear')  # Output layer for 2 variables
  ])
  
  # Step 4: Model Compilation and Training
  model.compile(optimizer='adam', loss='mean_squared_error')
  model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=32)
  model.save('trained_model.h5')
  
  # Step 5: Model Evaluation
  loss = model.evaluate(X_val, y_val)
  print(&quot;Validation Loss:&quot;, loss)
  </code></pre>
  <p>The randomness parameter impacts the way the voronoi shapes look. This image is generated by a high randomness parameter:</p>
  <p><a href="https://i.sstatic.net/4tSxJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4tSxJ.png" alt="enter image description here" /></a></p>
  <p>and this one with a low randomness parameter:</p>
  <p><a href="https://i.sstatic.net/WvhQh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WvhQh.png" alt="enter image description here" /></a></p>
  <p>Basically: 0 randomness = neat squares, 1 randomness = messy quads.</p>
  <p>Any suggestions on how to approach this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T09:07:46.577Z
Comments.Text: Could you give some details of your current model. "Using Keras" is not enough to understand what you are doing because Keras is a very generic and flexible neural network library. What kind of model did you make (layer types, number/size of layers), how did you prep the image data, how many training examples have you constructed? Are you fine-tuning an existing image classifier model, or training from scratch?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T12:48:17.767Z
Comments.Text: @NeilSlater - Thanks, I added the code for model generation, let me know if that desn't help.
Comments.UserDisplayName: simone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T12:59:24.830Z
Comments.Text: Thanks for the edit. Your model is much clearer. Other half of the issue is your training data set, and training process. What size is the dataset, how are you deciding on hyperparameters (have you run this with different models to search for best hyperparameters, or is this question following up on failed runs using this single model?)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T15:35:35.137Z
Comments.Text: @NeilSlater - the dataset is 1000 images. Since they are generated, I can generate as many as I want. As for parameters etc. this is very much cargo-cult: I am using ChatGPT to generate much of the code. As an explanation for this: I have tried going into the theory, but most of the examples bear little relation to the domain areas I am interested in, and this route has been more helpful in terms of giving me "aha - so this is what it's all about" moments . I plan to go back and revisit the theory in order to understand eand extend what i am doing
Comments.UserDisplayName: simone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T15:44:48.180Z
Comments.Text: I have enough for an answer, thanks, but one more thing is worth knowing (because I could skip it if it's a not issue). What is the numerical range of the scale factor that you are more successfully predicting with the model as-is?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T17:26:17.240Z
Comments.Text: @NeilSlater the range for scale is 10-100. In the meantime I've tried also keeping the scale fixed at 20 and kept the image at 2512x512, but got only marginal improvements (a correlation of 0.1876 and RMSE of 0.2928).
Comments.UserDisplayName: simone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T18:08:24.593Z
Comments.Text: OK, so having one element scaled 10-100, next to another at 0-1, with both scored simultaneously using RMSE error is going to be a problem. Relatively small errors with the scale value are going to produce large gradients in the rest of the model, swamping the gradient signal from the randomness param errors. You should normalise the params for NN output to put them into similar ranges and convert back later. This is not your only problem, but it is a significant one. There are also loss functions that can cope with different inherent scales better
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><computer-vision>
Title: Estimating texture parameters from images, one parameter is harder to learn
UserDisplayName: simone
ThreadText: # Estimating texture parameters from images, one parameter is harder to learn
  
  (Asked by: username_0 on 2023-08-24)
  
  I am trying to use Keras to estimate parameters used in the generation of Voronoi textures. I got good results in estimating the scale parameter, but extremely poor ones in the estimation of the randomness parameter (a correlation of 0.0551 and an RMSE of 0.5088 for a variable that ranges from 0 to 1).
  This is the model I am using:
  model = Sequential([
      Conv2D(32, (3, 3), activation='relu', input_shape=(256, 256, 1)),
      MaxPooling2D((2, 2)),
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D((2, 2)),
      Flatten(),
      Dense(128, activation='relu'),
      Dense(var_count, activation='linear')  # Output layer for 2 variables
  ])
  
  # Step 4: Model Compilation and Training
  model.compile(optimizer='adam', loss='mean_squared_error')
  model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=32)
  model.save('trained_model.h5')
  
  # Step 5: Model Evaluation
  loss = model.evaluate(X_val, y_val)
  print("Validation Loss:", loss)
  
  The randomness parameter impacts the way the voronoi shapes look. This image is generated by a high randomness parameter:
  
  and this one with a low randomness parameter:
  
  Basically: 0 randomness = neat squares, 1 randomness = messy quads.
  Any suggestions on how to approach this?
  
  **Tags:** machine-learning, computer-vision
  > **Comments:**
  > **username_1**: Could you give some details of your current model. "Using Keras" is not enough to understand what you are doing because Keras is a very generic and flexible neural network library. What kind of model did you make (layer types, number/size of layers), how did you prep the image data, how many training examples have you constructed? Are you fine-tuning an existing image classifier model, or training from scratch?  
  > **username_0**: @NeilSlater - Thanks, I added the code for model generation, let me know if that desn't help.  
  > **username_1**: Thanks for the edit. Your model is much clearer. Other half of the issue is your training data set, and training process. What size is the dataset, how are you deciding on hyperparameters (have you run this with different models to search for best hyperparameters, or is this question following up on failed runs using this single model?)  
  > **username_0**: @NeilSlater - the dataset is 1000 images. Since they are generated, I can generate as many as I want. As for parameters etc. this is very much cargo-cult: I am using ChatGPT to generate much of the code. As an explanation for this: I have tried going into the theory, but most of the examples bear little relation to the domain areas I am interested in, and this route has been more helpful in terms of giving me "aha - so this is what it's all about" moments . I plan to go back and revisit the theory in order to understand eand extend what i am doing  
  > **username_1**: I have enough for an answer, thanks, but one more thing is worth knowing (because I could skip it if it's a not issue). What is the numerical range of the scale factor that you are more successfully predicting with the model as-is?  
  > **username_0**: @NeilSlater the range for scale is 10-100. In the meantime I've tried also keeping the scale fixed at 20 and kept the image at 2512x512, but got only marginal improvements (a correlation of 0.1876 and RMSE of 0.2928).  
  > **username_1**: OK, so having one element scaled 10-100, next to another at 0-1, with both scored simultaneously using RMSE error is going to be a problem. Relatively small errors with the scale value are going to produce large gradients in the rest of the model, swamping the gradient signal from the randomness param errors. You should normalise the params for NN output to put them into similar ranges and convert back later. This is not your only problem, but it is a significant one. There are also loss functions that can cope with different inherent scales better  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-08-24. Score: 2 | Accepted answer
  
  The core issue here is that the two parameters you want to predict have actually different difficulty levels for a computer vision system. You have created a neural network model capable of solving the easier one, but not the more difficult one.
  
  The scale factor is relatively easy to estimate because it (inversely) correlates closely to the number of simple features in the image. A simple flexible corner detection filter, provided the scale is not too low, will when flattened produce an almost linear relationship based on counting the features. A very basic CNN, like the one in the question, should be capable of making this kind of calculation.
  
  The randomness factor is harder - it is going to be correlated in a more complex way to the *variance* of the corner and line features in the image. A variance is harder to calculate than a count of matches, so will likely require a more complex network, and more training examples.
  
  In addition, from comments, the scaling of the two parameters is radically different, which will cause a problem with processing the error signal. When predicting both values at once you should probably normalise the outputs so that one error signal does not dominate the neural network learning.
  
  Where to go from here? Here are a few suggestions:
  
  * Isolate the problem by only predicting the randomness variable. Then you can make the joint prediction of both paramaters as a later piece of work, once you have a better idea what works separately.
  * You need to perform a hyperparameter search - trying different neural network structures to see which perform better at the task. Initially I would recommend making a deeper fully-connected section (i.e. add a couple more `Dense(128,...` layers after the first one , but you can also play with the number of convolutional layers, and the number of features in each layer. More is not necessarily better, so you can try increasing number of layers, but decreasing the number of features or the number of virtual neurons.
    + Right now, whilst you are still learning, you can perform this search manually by editing your script, keeping notes on changes that seem to improve performance.
  * I would be concerned that 1000 training examples is not many for an image prediction task trained from scratch. Multiply by at least 10. Then you can check whether adding more examples improves the model by plotting a learning curve - train with 1000, 2000, 3000, 4000 etc, and see whether results improve. If results are still improving when moving from 9000 to 10000 training examples, it's worth going to 20000 total.
  
  
  ---
  
  ### Answer by username_2 on 2023-09-10. Score: 0
  
  Stacking ReLUs will cause "explosive" gradients, since their gains get multiplied for the outputs which are on the positive range. So I suggest that you add BatchNormalization layers there in-between. And because the dimension after the `Flatten` is 246k, the next Dense layer ends up having 31.5 million parameters! Using global average and global max pooling, I got a model with just 40k parameters to fit the data very nicely. But sure, our data generation codes aren't identical. Here are some sample images:
  
  [![vornoi with noise](https://i.sstatic.net/BEYmp.jpg)](https://i.sstatic.net/BEYmp.jpg)
  
  Here are scatter plots on predicted (x-axis) and correct (y-axis, with added noise) values. RMSE values are 0.08 and 0.17, so indeed the noise level was a bit more difficult to estimate. In the training data I scaled both parameters to the scale from 0 to 1.
  
  [![result scatter plots](https://i.sstatic.net/YVH2x.png)](https://i.sstatic.net/YVH2x.png)
  
  But the correlation-coefficient is so close to zero, that I started wondering whether you have mixed up your training data, and the target value was essentially random.
  
  > **Comments:**
  > **username_0**: that looks interesting - would you mind sharing some code? I can provide the code for training data if that's beneficial  
  > **username_2**: Here is my script: https://gist.github.com/nikonyrh/df50f1eb2490af7df685fa263ae5ad36 . If you are still having problems, I could try using your script to generate the data and see what is the accuracy.  
  > **username_0**: Awesome. I don't think I''l be able to test this before end of next week but I'll sure let you know  
  
  ---
  
  