Answers.Body: <p>According to my experience, it is possible to reach 99%+ accuracy on MNIST within a few epochs using a simple CNN. MNIST is really an easy dataset. So, it's likely that you've broken something as you're modifying author's code.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T18:31:28.263Z
Answers.Text: I didn't really "modify" much. The compiler couldn't find the to_categorical() or the Adadelta() functions in the imported libraries, so I imported tensorflow itself and prefixed the function call with it.
Answers.UserDisplayName: rad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T18:20:37.157Z
Answers.LastActivityDate: 2022-01-05T18:20:37.157Z
Answers.UserDisplayName: SpiderRico
Answers.Body: <p>I was able to run the code without &quot;any&quot; modifications on Tensorflow 2.4.0, just had to replace the imports:</p>
  <pre><code>import keras
  from keras.datasets import mnist
  ...
  </code></pre>
  <p>-&gt;</p>
  <pre><code>import tensorflow.keras as keras
  from tensorflow.keras.datasets import mnist
  ...
  </code></pre>
  <p>Output:</p>
  <pre><code>Epoch 1/12
  469/469 [==============================] - 4s 7ms/step - loss: 2.2914 - accuracy: 0.1260 - val_loss: 2.2425 - val_accuracy: 0.2915
  Epoch 2/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.2375 - accuracy: 0.2383 - val_loss: 2.1710 - val_accuracy: 0.4693
  Epoch 3/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.1665 - accuracy: 0.3439 - val_loss: 2.0758 - val_accuracy: 0.5645
  Epoch 4/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.0750 - accuracy: 0.4286 - val_loss: 1.9489 - val_accuracy: 0.6155
  Epoch 5/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.9510 - accuracy: 0.4964 - val_loss: 1.7919 - val_accuracy: 0.6633
  Epoch 6/12
  469/469 [==============================] - 4s 8ms/step - loss: 1.8107 - accuracy: 0.5401 - val_loss: 1.6096 - val_accuracy: 0.7082
  Epoch 7/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.6469 - accuracy: 0.5737 - val_loss: 1.4162 - val_accuracy: 0.7475
  Epoch 8/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.4888 - accuracy: 0.6095 - val_loss: 1.2314 - val_accuracy: 0.7727
  Epoch 9/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.3350 - accuracy: 0.6409 - val_loss: 1.0702 - val_accuracy: 0.7961
  Epoch 10/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.2118 - accuracy: 0.6651 - val_loss: 0.9390 - val_accuracy: 0.8161
  Epoch 11/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.1072 - accuracy: 0.6855 - val_loss: 0.8373 - val_accuracy: 0.8291
  Epoch 12/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.0208 - accuracy: 0.7061 - val_loss: 0.7568 - val_accuracy: 0.8370
  
  Test loss: 0.7567713856697083
  Test accuracy: 0.8370000123977661
  </code></pre>
  <p>Running it 10 times (it took about 30 seconds / model on 1080 Ti):</p>
  <pre><code>Test loss: 0.71385968 +/- 0.0471146
  Test accuracy: 0.84305001 +/- 0.00868576
  </code></pre>
  <p>So clearly these are very different results than the author got, but also significantly higher than what You got. He might have used different parameters with the Adadelta optimizer, or a different version of it.</p>
  <p>Usually I've had the best results with Adam (default parameters, learning rate = 0.001). I tested running it 5 times with these results:</p>
  <pre><code>Test loss: 0.02858894 +/- 0.00151724
  Test accuracy: 0.99186 +/- 0.00058172
  </code></pre>
  <p>Adam is 3 years newer than Adadelta (2014 vs. 2011).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T21:08:44.630Z
Answers.Text: Thank you so much! This really clears things up. I think 0.6 was the lowest I got. My more recent ones were in the same range as yours. It was just the instant high accuracy and low loss from his data that was really throwing me off. However, I'm able to mimic that using Adam like you said. Happy to see everything's running as its supposed to. Guess I'll stick to Adam for now.
Answers.UserDisplayName: rad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T20:13:28.393Z
Answers.LastActivityDate: 2022-01-05T20:13:28.393Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I've been following a deep learning book and the current section I'm on is about convolutional neural networks. The author presents some code to create a basic CNN with about 1 million parameters, which he manages to train to 99.2% accuracy within 12 epochs on the full MNIST dataset.</p>
  <p>His output looks like this:</p>
  <pre><code>Train on 60000 samples, validate on 10000 samples 
  Epoch 1/12-loss:0.2800 acc:0.9147 val_loss:0.0624 val_acc:0.9794 
  Epoch 2/12-loss:0.1003 acc:0.9695 val_loss:0.0422 val_acc:0.9854 
  Epoch 3/12-loss:0.0697 acc:0.9789 val_loss:0.0356 val_acc:0.9880 
  Epoch 4/12-loss:0.0573 acc:0.9827 val_loss:0.0282 val_acc:0.9910 
  Epoch 5/12-loss:0.0478 acc:0.9854 val_loss:0.0311 val_acc:0.9901 
  Epoch 6/12-loss:0.0419 acc:0.9871 val_loss:0.0279 val_acc:0.9908 
  Epoch 7/12-loss:0.0397 acc:0.9883 val_loss:0.0250 val_acc:0.9914 
  Epoch 8/12-loss:0.0344 acc:0.9891 val_loss:0.0288 val_acc:0.9910 
  Epoch 9/12-loss:0.0329 acc:0.9895 val_loss:0.0273 val_acc:0.9916 
  Epoch 10/12-loss:0.0305 acc:0.9909 val_loss:0.0296 val_acc:0.9904 
  Epoch 11/12-loss:0.0291 acc:0.9911 val_loss:0.0275 val_acc:0.9920 
  Epoch 12/12-loss:0.0274 acc:0.9916 val_loss:0.0245 val_acc:0.9916 
  Test loss:
  0.02452171179684301 Test accuracy: 0.9916
  </code></pre>
  <p>Using <a href="https://github.com/rkneusel9/PracticalDeepLearningPython/blob/main/chapter_13/mnist_cnn_exp0.py" rel="nofollow noreferrer">this</a> code.</p>
  <p>Running that same code on my machine, after 12 epochs, I'm barely at 0.6 accuracy. I did have to modify a couple function calls as keras has changed a couple things since the book came out. I'm going crazy trying to figure out why his is so quick. He presents the results as &quot;this is what should happen when the code is run.&quot; I understand it had already went through almost 500 gradient descent steps by the time epoch 1 is complete, but is that really enough to reach 98% accuracy right off the bat??</p>
  
Tags: <convolutional-neural-networks><tensorflow><python><keras>
Title: Are these book example CNN results realistic?
UserDisplayName: rad
ThreadText: # Are these book example CNN results realistic?
  
  (Asked by: username_0 on 2022-01-05)
  
  I've been following a deep learning book and the current section I'm on is about convolutional neural networks. The author presents some code to create a basic CNN with about 1 million parameters, which he manages to train to 99.2% accuracy within 12 epochs on the full MNIST dataset.
  His output looks like this:
  Train on 60000 samples, validate on 10000 samples 
  Epoch 1/12-loss:0.2800 acc:0.9147 val_loss:0.0624 val_acc:0.9794 
  Epoch 2/12-loss:0.1003 acc:0.9695 val_loss:0.0422 val_acc:0.9854 
  Epoch 3/12-loss:0.0697 acc:0.9789 val_loss:0.0356 val_acc:0.9880 
  Epoch 4/12-loss:0.0573 acc:0.9827 val_loss:0.0282 val_acc:0.9910 
  Epoch 5/12-loss:0.0478 acc:0.9854 val_loss:0.0311 val_acc:0.9901 
  Epoch 6/12-loss:0.0419 acc:0.9871 val_loss:0.0279 val_acc:0.9908 
  Epoch 7/12-loss:0.0397 acc:0.9883 val_loss:0.0250 val_acc:0.9914 
  Epoch 8/12-loss:0.0344 acc:0.9891 val_loss:0.0288 val_acc:0.9910 
  Epoch 9/12-loss:0.0329 acc:0.9895 val_loss:0.0273 val_acc:0.9916 
  Epoch 10/12-loss:0.0305 acc:0.9909 val_loss:0.0296 val_acc:0.9904 
  Epoch 11/12-loss:0.0291 acc:0.9911 val_loss:0.0275 val_acc:0.9920 
  Epoch 12/12-loss:0.0274 acc:0.9916 val_loss:0.0245 val_acc:0.9916 
  Test loss:
  0.02452171179684301 Test accuracy: 0.9916
  
  Using this code.
  Running that same code on my machine, after 12 epochs, I'm barely at 0.6 accuracy. I did have to modify a couple function calls as keras has changed a couple things since the book came out. I'm going crazy trying to figure out why his is so quick. He presents the results as "this is what should happen when the code is run." I understand it had already went through almost 500 gradient descent steps by the time epoch 1 is complete, but is that really enough to reach 98% accuracy right off the bat??
  
  **Tags:** convolutional-neural-networks, tensorflow, python, keras
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-01-05. Score: 3 | Accepted answer
  
  I was able to run the code without "any" modifications on Tensorflow 2.4.0, just had to replace the imports:
  import keras
  from keras.datasets import mnist
  ...
  
  ->
  import tensorflow.keras as keras
  from tensorflow.keras.datasets import mnist
  ...
  
  Output:
  Epoch 1/12
  469/469 [==============================] - 4s 7ms/step - loss: 2.2914 - accuracy: 0.1260 - val_loss: 2.2425 - val_accuracy: 0.2915
  Epoch 2/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.2375 - accuracy: 0.2383 - val_loss: 2.1710 - val_accuracy: 0.4693
  Epoch 3/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.1665 - accuracy: 0.3439 - val_loss: 2.0758 - val_accuracy: 0.5645
  Epoch 4/12
  469/469 [==============================] - 3s 7ms/step - loss: 2.0750 - accuracy: 0.4286 - val_loss: 1.9489 - val_accuracy: 0.6155
  Epoch 5/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.9510 - accuracy: 0.4964 - val_loss: 1.7919 - val_accuracy: 0.6633
  Epoch 6/12
  469/469 [==============================] - 4s 8ms/step - loss: 1.8107 - accuracy: 0.5401 - val_loss: 1.6096 - val_accuracy: 0.7082
  Epoch 7/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.6469 - accuracy: 0.5737 - val_loss: 1.4162 - val_accuracy: 0.7475
  Epoch 8/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.4888 - accuracy: 0.6095 - val_loss: 1.2314 - val_accuracy: 0.7727
  Epoch 9/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.3350 - accuracy: 0.6409 - val_loss: 1.0702 - val_accuracy: 0.7961
  Epoch 10/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.2118 - accuracy: 0.6651 - val_loss: 0.9390 - val_accuracy: 0.8161
  Epoch 11/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.1072 - accuracy: 0.6855 - val_loss: 0.8373 - val_accuracy: 0.8291
  Epoch 12/12
  469/469 [==============================] - 3s 7ms/step - loss: 1.0208 - accuracy: 0.7061 - val_loss: 0.7568 - val_accuracy: 0.8370
  
  Test loss: 0.7567713856697083
  Test accuracy: 0.8370000123977661
  
  Running it 10 times (it took about 30 seconds / model on 1080 Ti):
  Test loss: 0.71385968 +/- 0.0471146
  Test accuracy: 0.84305001 +/- 0.00868576
  
  So clearly these are very different results than the author got, but also significantly higher than what You got. He might have used different parameters with the Adadelta optimizer, or a different version of it.
  Usually I've had the best results with Adam (default parameters, learning rate = 0.001). I tested running it 5 times with these results:
  Test loss: 0.02858894 +/- 0.00151724
  Test accuracy: 0.99186 +/- 0.00058172
  
  Adam is 3 years newer than Adadelta (2014 vs. 2011).
  
  > **Comments:**
  > **username_0**: Thank you so much! This really clears things up. I think 0.6 was the lowest I got. My more recent ones were in the same range as yours. It was just the instant high accuracy and low loss from his data that was really throwing me off. However, I'm able to mimic that using Adam like you said. Happy to see everything's running as its supposed to. Guess I'll stick to Adam for now.  
  
  ---
  
  ### Answer by username_1 on 2022-01-05. Score: 0
  
  According to my experience, it is possible to reach 99%+ accuracy on MNIST within a few epochs using a simple CNN. MNIST is really an easy dataset. So, it's likely that you've broken something as you're modifying author's code.
  
  > **Comments:**
  > **username_0**: I didn't really "modify" much. The compiler couldn't find the to_categorical() or the Adadelta() functions in the imported libraries, so I imported tensorflow itself and prefixed the function call with it.  
  
  ---
  
  