Answers.Body: <p>It is a difficult task to identify the areas to be corrected to improve accuracy unless that area is looking into your face. By that I mean unless the regularizing parameter has unusually large or small values it is difficult to pin down and identify which regularizer to tweak. You need to do a grid search or random search over the hyper-parameter space to come up with an optimal combination of hyper parameters. </p>
  
  <p>There could be a lot of things that you could consider to reduce over fitting. Some of them are <span class="math-container">$L_2$</span> regularizers, Dropout, depth of the network, number of neurons in the layers, the optimizer,  etc. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-20T07:45:52.087Z
Answers.LastActivityDate: 2019-04-20T10:01:54.327Z
Answers.LastEditDate: 2019-04-20T10:01:54.327Z
Answers.UserDisplayName: naive
Body: <p>I am trying classify CIFAR10. The CNN that I generated over fits when the accuracy reaches ~77%. The code and the plot is given below. I tried DropOut, Batch Normalization and L2 Regularization. But the accuracy does not go beyond ~77.  </p>
  
  <p>How can I identify the areas to be corrected to reduce over fitting? </p>
  
  <pre><code>convolutional_model = Sequential()
  
  # 32
  convolutional_model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3), kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  # 64
  convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(Conv2D(128, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  convolutional_model.add(Flatten())
  convolutional_model.add(Dropout(0.5))
  
  convolutional_model.add(Dense(128, activation='relu'))
  convolutional_model.add(Dense(10, activation='softmax'))
  
  print(convolutional_model.summary())
  convolutional_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  es = EarlyStopping(monitor='val_loss', mode='min', verbose=2, patience=8)
  history = convolutional_model.fit(X_Train_Part, Y_Train_Part, epochs=200, verbose=2,validation_data=(X_Train_Validate, Y_Train_Validate), callbacks=[es])
  
  scores = convolutional_model.evaluate(X_Test, Y_Test, verbose=2)
  </code></pre>
  
  <p><a href="https://i.sstatic.net/TAOQ5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TAOQ5.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-19T18:25:08.950Z
Comments.Text: Please, ask just one question per post!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-19T19:32:35.267Z
Comments.Text: Welcome to SE:AI!  I'm leaving this question open pending more closevotes, but please consider reducing it to a single question.  (You can always ask additional question in separate posts.)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-20T01:40:47.927Z
Comments.Text: Edited the post to reduce number of questions
Comments.UserDisplayName: Maanu
Tags: <deep-learning><convolutional-neural-networks><overfitting>
Title: How to identify the areas to reduce over fitting?
UserDisplayName: Maanu
ThreadText: # How to identify the areas to reduce over fitting?
  
  (Asked by: username_0 on 2019-04-19)
  
  I am trying classify CIFAR10. The CNN that I generated over fits when the accuracy reaches ~77%. The code and the plot is given below. I tried DropOut, Batch Normalization and L2 Regularization. But the accuracy does not go beyond ~77.
  
  How can I identify the areas to be corrected to reduce over fitting?
  
  ```
  convolutional_model = Sequential()
  
  # 32
  convolutional_model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3), kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  # 64
  convolutional_model.add(Conv2D(64, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(Conv2D(128, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(.0002)))
  convolutional_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
  
  convolutional_model.add(Flatten())
  convolutional_model.add(Dropout(0.5))
  
  convolutional_model.add(Dense(128, activation='relu'))
  convolutional_model.add(Dense(10, activation='softmax'))
  
  print(convolutional_model.summary())
  convolutional_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  es = EarlyStopping(monitor='val_loss', mode='min', verbose=2, patience=8)
  history = convolutional_model.fit(X_Train_Part, Y_Train_Part, epochs=200, verbose=2,validation_data=(X_Train_Validate, Y_Train_Validate), callbacks=[es])
  
  scores = convolutional_model.evaluate(X_Test, Y_Test, verbose=2)
  
  ```
  
  [![enter image description here](https://i.sstatic.net/TAOQ5.png)](https://i.sstatic.net/TAOQ5.png)
  
  **Tags:** deep-learning, convolutional-neural-networks, overfitting
  
  > **Comments:**
  > **username_1**: Please, ask just one question per post!  
  > **username_2**: Welcome to SE:AI!  I'm leaving this question open pending more closevotes, but please consider reducing it to a single question.  (You can always ask additional question in separate posts.)  
  > **username_0**: Edited the post to reduce number of questions  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-04-20. Score: 1
  
  It is a difficult task to identify the areas to be corrected to improve accuracy unless that area is looking into your face. By that I mean unless the regularizing parameter has unusually large or small values it is difficult to pin down and identify which regularizer to tweak. You need to do a grid search or random search over the hyper-parameter space to come up with an optimal combination of hyper parameters.
  
  There could be a lot of things that you could consider to reduce over fitting. Some of them are $L\_2$ regularizers, Dropout, depth of the network, number of neurons in the layers, the optimizer, etc.
  
  
  ---
  
  