Answers.Body: <p>Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions. </p>
  
  <p>In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.</p>
  
  <p>So if you had <code>Conv -&gt; pool -&gt; Conv -&gt; pool -&gt; ...</code> , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.</p>
  
  <p>In your network you start with a 28*28 image, and you use 32 3*3 filters. so number of parameters is <code>(3*3 + 1) * 32 = 320</code>.</p>
  
  <p>In the dense layer you have as input a <code>13*13*32</code> and use a 100 FC layer. so n_parameters is <code>(13*13*32 + 1)*100</code> which is <code>540900</code>.</p>
  
  <p>Then you get <code>(100+1) * 10</code> FC which is <code>1010</code> more.</p>
  
  <p><code>total = 320 + 540900 + 1000</code> which is <code>542230</code>, as expected.</p>
  
  <p>The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.</p>
  
  <p>Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-26T20:27:19.153Z
Answers.LastActivityDate: 2019-12-26T20:32:30.577Z
Answers.LastEditDate: 2019-12-26T20:32:30.577Z
Answers.UserDisplayName: Miguel Saraiva
Body: <p>Okay so here's my CNN (simple example from a tutorial) along with some arithmetic to get the total number of free parameters.</p>
  
  <p>We've got a dataset of 28*28 grayscale image (MNIST).</p>
  
  <ol>
  <li>First layer is a 2D convolution using 32 3x3 kernels. Dimensionality of the output is 26x26x32 (kernel stride length was 1 and we have 32 feature maps of 26x26). <strong>Running parameter count: 288</strong></li>
  <li>Second layer is 2x2 MaxPool with a 2x2. Dimensionality of the output is 13x13x32 but then we flatten so we got a vector of length 5408. No extra parameters here.</li>
  <li>Third layer is Dense. A 5408x100 matrix. Dimensionality of the output is 100. <strong>Running Parameter count: 540988</strong></li>
  <li>Fourth layer is Dense also. A 100x10 matrix. Dimensionality of the output is 10. <strong>Running Parameter count: 541988</strong></li>
  </ol>
  
  <p>Then we're supposed to do stochastic gradient descent on a 541988 parameter space!</p>
  
  <p>That feels like a ridiculously big number to me. And this is meant to be the hello world problem of CNNs. Am I missing something fundamental in my understanding of how this is meant to work? Or maybe the number is correct but it's not actually a big deal for a computer to crunch?</p>
  
  <p>In case it helps. Here is how the model was built in Keras:</p>
  
  <pre><code>def define_model():
      model = Sequential()
      model.add(Conv2D(32, (3,3), activation = 'relu', kernel_initializer = 'he_uniform', input_shape=(28,28,1)))
      model.add(MaxPooling2D((2,2)))
      model.add(Flatten())
      model.add(Dense(100, activation='relu', kernel_initializer='he_uniform'))
      model.add(Dense(10, activation='softmax'))
      opt = SGD(lr=0.01, momentum=0.9)
      model.compile(optimizer=opt, loss='categorical_crossentropy', metric=['accuracy'])
      return model
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T19:33:11.320Z
Comments.Text: I do not think the number is that high, but you can always lower it by making 1 more convolution early for example
Comments.UserDisplayName: Miguel Saraiva
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T19:40:12.787Z
Comments.Text: Is there something wrong in my working? Also, wouldn't adding another convolution simply increase the number of free parameters?
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T20:01:27.100Z
Comments.Text: Depends on the kernel size, if you use a bigger one without padding no. I do not see anything wrong with it but I'm no expert. If you are worried about the number of parameters probably adding another pair Conv2D + Pool(2,2) will reduce them. Although it adds a new layer with parameters it will reduce the size of ur current third layer by a lot in terms of number of parameters.
Comments.UserDisplayName: Miguel Saraiva
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T20:12:29.520Z
Comments.Text: Thanks @MiguelSaraiva. I'm not so worried about reducing the dimensionality. I was more interested in consolidating my understanding of the optimisation problem. I think it's about right. I used `print(model.count_params())` and got 542230. I just need to figure out where the extra 242 parameters came from.
Comments.UserDisplayName: Alexander Soare
Tags: <convolutional-neural-networks><python><keras><gradient-descent>
Title: How many parameters are being optimised over in a simple CNN?
UserDisplayName: Alexander Soare
ThreadText: # How many parameters are being optimised over in a simple CNN?
  
  (Asked by: username_0 on 2019-12-26)
  
  Okay so here's my CNN (simple example from a tutorial) along with some arithmetic to get the total number of free parameters.
  We've got a dataset of 28*28 grayscale image (MNIST).
  
  First layer is a 2D convolution using 32 3x3 kernels. Dimensionality of the output is 26x26x32 (kernel stride length was 1 and we have 32 feature maps of 26x26). Running parameter count: 288
  Second layer is 2x2 MaxPool with a 2x2. Dimensionality of the output is 13x13x32 but then we flatten so we got a vector of length 5408. No extra parameters here.
  Third layer is Dense. A 5408x100 matrix. Dimensionality of the output is 100. Running Parameter count: 540988
  Fourth layer is Dense also. A 100x10 matrix. Dimensionality of the output is 10. Running Parameter count: 541988
  
  Then we're supposed to do stochastic gradient descent on a 541988 parameter space!
  That feels like a ridiculously big number to me. And this is meant to be the hello world problem of CNNs. Am I missing something fundamental in my understanding of how this is meant to work? Or maybe the number is correct but it's not actually a big deal for a computer to crunch?
  In case it helps. Here is how the model was built in Keras:
  def define_model():
      model = Sequential()
      model.add(Conv2D(32, (3,3), activation = 'relu', kernel_initializer = 'he_uniform', input_shape=(28,28,1)))
      model.add(MaxPooling2D((2,2)))
      model.add(Flatten())
      model.add(Dense(100, activation='relu', kernel_initializer='he_uniform'))
      model.add(Dense(10, activation='softmax'))
      opt = SGD(lr=0.01, momentum=0.9)
      model.compile(optimizer=opt, loss='categorical_crossentropy', metric=['accuracy'])
      return model
  
  **Tags:** convolutional-neural-networks, python, keras, gradient-descent
  > **Comments:**
  > **username_1**: I do not think the number is that high, but you can always lower it by making 1 more convolution early for example  
  > **username_0**: Is there something wrong in my working? Also, wouldn't adding another convolution simply increase the number of free parameters?  
  > **username_1**: Depends on the kernel size, if you use a bigger one without padding no. I do not see anything wrong with it but I'm no expert. If you are worried about the number of parameters probably adding another pair Conv2D + Pool(2,2) will reduce them. Although it adds a new layer with parameters it will reduce the size of ur current third layer by a lot in terms of number of parameters.  
  > **username_0**: Thanks @MiguelSaraiva. I'm not so worried about reducing the dimensionality. I was more interested in consolidating my understanding of the optimisation problem. I think it's about right. I used `print(model.count_params())` and got 542230. I just need to figure out where the extra 242 parameters came from.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-26. Score: 2 | Accepted answer
  
  Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions.
  
  In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.
  
  So if you had `Conv -> pool -> Conv -> pool -> ...` , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.
  
  In your network you start with a 28\*28 image, and you use 32 3\*3 filters. so number of parameters is `(3*3 + 1) * 32 = 320`.
  
  In the dense layer you have as input a `13*13*32` and use a 100 FC layer. so n\_parameters is `(13*13*32 + 1)*100` which is `540900`.
  
  Then you get `(100+1) * 10` FC which is `1010` more.
  
  `total = 320 + 540900 + 1000` which is `542230`, as expected.
  
  The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.
  
  Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.
  
  
  ---
  
  