Body: <p>After <a href="https://stackoverflow.com/q/77607526/15459840">asking on StackOverflow</a>, I was redirected here, so I'm reposting this question.</p>
  <p>I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try and use some of what I've learned for a problem I'm having. After some studying, I've understood how to build a Neural Network for my purpose, but I can't find relevant info about how to build a <em>good</em> Neural Network apart from the good old <a href="https://www.urbandictionary.com/define.php?term=Fuck%20Around%20And%20Find%20Out" rel="nofollow noreferrer">trial and error</a>. Here I attach the NN I'm currently working with as an example, but my question applies to the general case of a (regression) neural network: <strong>is there some theory on why I should build an architecture instead of another one, what activator I should choose, why I should lower my learning rate and how much, why should my dropout rate be higher and how much, how much training data is enough</strong>, and all these sorts of things?</p>
  <p>My NN takes as input a 2x7 array of real values in [0,1] and gives as output a single real value, and it looks like this:</p>
  
  <pre><code>model_cnn = Sequential()
  model_cnn.add(Conv2D(32, (2, 2), activation='relu', input_shape=(2, 7, 1), padding='same', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Conv2D(64, (2, 2), activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Flatten())
  model_cnn.add(Dropout(0.5)) 
  model_cnn.add(Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Dropout(0.5)) 
  model_cnn.add(Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Dense(1, activation='linear')) #linear for regression
  
  def lr_schedule(epoch):
      lr = 1e-3
      if epoch &gt; 50:
          lr *= 0.1
      if epoch &gt; 100:
          lr *= 0.1
      return lr
  
  lr_scheduler = keras.callbacks.LearningRateScheduler(lr_schedule)
  early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', 
                          patience=20, 
                          restore_best_weights=True)
  
  model_cnn.compile(loss='mean_squared_error',
                optimizer=keras.optimizers.Adam(learning_rate=lr_schedule(0)),
                metrics=['mean_absolute_error'])
  
  nn_history = model_cnn.fit(X_train, y_train,
                          batch_size=64,
                          epochs=1000,
                          verbose=1,
                          validation_data=(X_val, y_val),
                          callbacks=[lr_scheduler, early_stopping])
  </code></pre>
  <p>This is the result of some adjustments, for example adding dropout and normalization, that I did just by feeling, without any actual knowledge of, for example, whether it is correct to put them in the above order. Again: I know I can just try to change it and see what happens, but I'm asking how (if there is a way!) to decide what are the plausible, if not the best, things to do.<br />
  As of now, the loss-vs-epoch looks like this:</p>
  <p><a href="https://i.sstatic.net/CIzMd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CIzMd.png" alt="enter image description here" /></a></p>
  <p>Doesn't look too bad, but I would like the loss to converge to zero (or at least to a value closer to zero). How can I understand what things are &quot;worth trying&quot;?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-06T21:43:44.427Z
Comments.Text: Domanda interessante. Ti ho votato. Seguo la domanda
Comments.UserDisplayName: Zollikofen4
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-07T13:49:29.093Z
Comments.Text: To get reasonable answer to this question(s), you should read a machine learning book like Hands-On Machine Learning Geron Aurelien; pattern recognition and machine learning/ Deep Learning Foundations and Concepts by bishop. bishop books are somehow hard to read, but Aurelien's book is easy. There are chapters in this book that are discussing and answering each part of your question in depth. Here is my concise answer: although it may seem to be trial, but designing a good architecture requires in depth understanding of many concept, Style GANs weren't obtained by heuristic
Comments.UserDisplayName: Reza_va
Tags: <neural-networks><convolutional-neural-networks><python><convergence>
Title: Is improving a Neural Network really just "trial and error"?
UserDisplayName: Mauro Giliberti
ThreadText: # Is improving a Neural Network really just "trial and error"?
  
  (Asked by: username_0 on 2023-12-06)
  
  After asking on StackOverflow, I was redirected here, so I'm reposting this question.
  I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try and use some of what I've learned for a problem I'm having. After some studying, I've understood how to build a Neural Network for my purpose, but I can't find relevant info about how to build a good Neural Network apart from the good old trial and error. Here I attach the NN I'm currently working with as an example, but my question applies to the general case of a (regression) neural network: is there some theory on why I should build an architecture instead of another one, what activator I should choose, why I should lower my learning rate and how much, why should my dropout rate be higher and how much, how much training data is enough, and all these sorts of things?
  My NN takes as input a 2x7 array of real values in [0,1] and gives as output a single real value, and it looks like this:
  model_cnn = Sequential()
  model_cnn.add(Conv2D(32, (2, 2), activation='relu', input_shape=(2, 7, 1), padding='same', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Conv2D(64, (2, 2), activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Flatten())
  model_cnn.add(Dropout(0.5)) 
  model_cnn.add(Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Dropout(0.5)) 
  model_cnn.add(Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
  model_cnn.add(BatchNormalization())
  model_cnn.add(Dense(1, activation='linear')) #linear for regression
  
  def lr_schedule(epoch):
      lr = 1e-3
      if epoch > 50:
          lr *= 0.1
      if epoch > 100:
          lr *= 0.1
      return lr
  
  lr_scheduler = keras.callbacks.LearningRateScheduler(lr_schedule)
  early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', 
                          patience=20, 
                          restore_best_weights=True)
  
  model_cnn.compile(loss='mean_squared_error',
                optimizer=keras.optimizers.Adam(learning_rate=lr_schedule(0)),
                metrics=['mean_absolute_error'])
  
  nn_history = model_cnn.fit(X_train, y_train,
                          batch_size=64,
                          epochs=1000,
                          verbose=1,
                          validation_data=(X_val, y_val),
                          callbacks=[lr_scheduler, early_stopping])
  
  This is the result of some adjustments, for example adding dropout and normalization, that I did just by feeling, without any actual knowledge of, for example, whether it is correct to put them in the above order. Again: I know I can just try to change it and see what happens, but I'm asking how (if there is a way!) to decide what are the plausible, if not the best, things to do.
  As of now, the loss-vs-epoch looks like this:
  
  Doesn't look too bad, but I would like the loss to converge to zero (or at least to a value closer to zero). How can I understand what things are "worth trying"?
  
  **Tags:** neural-networks, convolutional-neural-networks, python, convergence
  > **Comments:**
  > **username_1**: Domanda interessante. Ti ho votato. Seguo la domanda  
  > **username_2**: To get reasonable answer to this question(s), you should read a machine learning book like Hands-On Machine Learning Geron Aurelien; pattern recognition and machine learning/ Deep Learning Foundations and Concepts by bishop. bishop books are somehow hard to read, but Aurelien's book is easy. There are chapters in this book that are discussing and answering each part of your question in depth. Here is my concise answer: although it may seem to be trial, but designing a good architecture requires in depth understanding of many concept, Style GANs weren't obtained by heuristic  
  
  ---
  
  ## 0 answers
  
  