Answers.Body: <p>If I understood correctly you want to decrease the parameters count on the last layer (<code>dense_2</code> layer right?). It would be nice to know why you want to decrease the number of parameters in the last layers... But I'll proceed with what I see.</p>
  
  <p>Firstly, the dense layers (or fully connected in literature) have a deterministic number of parameters (or weights) to learn according to the size of the input and output tensor. The relation is the following:</p>
  
  <p><span class="math-container">$N_{params} = Y_{output} \cdot (X_{input} +1) = 200 \cdot (2209 +1) = 442000$</span></p>
  
  <p>Where:</p>
  
  <ul>
  <li><span class="math-container">$Y_{output}$</span>: Output tensor shape (<code>act2=200</code> which is the output of <code>dense_2</code> layer)</li>
  <li><span class="math-container">$X_{input}$</span>: Input tensor shape (<code>x=2209</code>which is the ouput of <code>flatten</code> layer)</li>
  </ul>
  
  <p>So if you want to decrease the number of parameters you can:</p>
  
  <ul>
  <li>Decrease the input tensor: your <code>self.action2</code> which I guess is the action space so you might not be able to decrease it</li>
  <li>Decrease the output tensor: maybe? I would need more context (code) to know if that is even possible</li>
  </ul>
  
  <p>So, in short: if you are not willing to change your the input or output tensor to the Dense layers, then no, you can not decrease the number of parameters.</p>
  
  <hr>
  
  <p><strong>BONUS</strong>: In case you missed it, I have noticed you have set your dense layers to <code>trainable=False</code>. So in principle you should not care about decreasing number of parameters (which in most cases is motivated to reduce training time) since they are already not being trained. You can check that in the Keras summary output:</p>
  
  <pre><code>Total params: 4,869,857
  Trainable params: 7,857
  Non-trainable params: 4,862,000
  </code></pre>
  
  <p>Where the non-trainable parameters are <span class="math-container">$4862000 = 4420000 + 442000 $</span>, which are the number of parameters of your 2 dense layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T21:47:40.730Z
Answers.Text: If I froze the action layer, would it degrade performance? Likewise, if I bottlenecked the layer above the action layers, wouldn't there be a loss of information?
Answers.UserDisplayName: ZeroMaxinumXZ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T09:50:42.560Z
Answers.Text: What is the action layer you are referring to?... I mean, you already froze the 2 dense layers which outputs `act1` and `act2` by setting `trainable=False`. Why did you freeze them? I you posted more code I would be able to get the context of it.
Answers.UserDisplayName: JVGD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T13:50:55.170Z
Answers.LastActivityDate: 2020-01-28T13:50:55.170Z
Answers.UserDisplayName: JVGD
Body: <p>Not sure where to put this... I am trying to create a convolutional architecture for a DQN in keras, and I want to know why my param count is so high for my last layer compared to the rest of the network. I've tried slowly decreasing the dimensions of the layers above it, but it performs quite poorly. I want to know if there's anything I can do to decrease the param count of that last layer, besides the above.</p>
  
  <p>Code:</p>
  
  <pre><code>#Import statements.
  import random
  import numpy as np
  import tensorflow as tf
  import tensorflow.keras.layers as L
  from collections import deque
  import layers as mL
  import tensorflow.keras.optimizers as O
  import optimizers as mO
  import tensorflow.keras.backend as K
  
  
  #Conv function.
  def conv(x, units, kernel, stride, noise=False, padding='valid'):
      y = L.Conv2D(units, kernel, stride, activation=mish, padding=padding)(x)
      if noise:
          y = mL.PGaussian()(y)
      return y
  
  #Network
          x_input = L.Input(shape=self.state)
          x_goal = L.Input(shape=self.state)
          x = L.Concatenate(-1)([x_input, x_goal])
          x_list = []
          for i in range(2):
              x = conv(x, 4, (7,7), 1)
          for i in range(2):
              x = conv(x, 8, (5,5), 2)
          for i in range(10):
              x = conv(x, 6, (3,3), 1, noise=True)
          x = L.Conv2D(1, (3,3), 1)(x)
          x_shape = K.int_shape(x)
          x = L.Reshape((x_shape[1], x_shape[2]))(x)
          x = L.Flatten()(x)
          crit = L.Dense(1, trainable=False)(x)
          critic = tf.keras.models.Model([x_input, x_goal], crit)
          act1 = L.Dense(self.action, trainable=False)(x)
          act2 = L.Dense(self.action2, trainable=False)(x)
          act1 = L.Softmax()(act1)
          act2 = L.Softmax()(act2)
          actor = tf.keras.models.Model([x_input, x_goal], [act1, act2])
          actor.compile(loss=mish_loss, optimizer='adam')
          actor.summary()
  
  </code></pre>
  
  <p>actor.summary():</p>
  
  <pre><code>________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_2 (InputLayer)            [(None, 300, 300, 3) 0                                            
  __________________________________________________________________________________________________
  input_3 (InputLayer)            [(None, 300, 300, 3) 0                                            
  __________________________________________________________________________________________________
  concatenate (Concatenate)       (None, 300, 300, 6)  0           input_2[0][0]                    
                                                                   input_3[0][0]                    
  __________________________________________________________________________________________________
  conv2d_52 (Conv2D)              (None, 294, 294, 4)  1180        concatenate[0][0]                
  __________________________________________________________________________________________________
  conv2d_53 (Conv2D)              (None, 288, 288, 4)  788         conv2d_52[0][0]                  
  __________________________________________________________________________________________________
  conv2d_54 (Conv2D)              (None, 142, 142, 8)  808         conv2d_53[0][0]                  
  __________________________________________________________________________________________________
  conv2d_55 (Conv2D)              (None, 69, 69, 8)    1608        conv2d_54[0][0]                  
  __________________________________________________________________________________________________
  conv2d_56 (Conv2D)              (None, 67, 67, 6)    438         conv2d_55[0][0]                  
  __________________________________________________________________________________________________
  p_gaussian (PGaussian)          (None, 67, 67, 6)    1           conv2d_56[0][0]                  
  __________________________________________________________________________________________________
  conv2d_57 (Conv2D)              (None, 65, 65, 6)    330         p_gaussian[0][0]                 
  __________________________________________________________________________________________________
  p_gaussian_1 (PGaussian)        (None, 65, 65, 6)    1           conv2d_57[0][0]                  
  __________________________________________________________________________________________________
  conv2d_58 (Conv2D)              (None, 63, 63, 6)    330         p_gaussian_1[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_2 (PGaussian)        (None, 63, 63, 6)    1           conv2d_58[0][0]                  
  __________________________________________________________________________________________________
  conv2d_59 (Conv2D)              (None, 61, 61, 6)    330         p_gaussian_2[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_3 (PGaussian)        (None, 61, 61, 6)    1           conv2d_59[0][0]                  
  __________________________________________________________________________________________________
  conv2d_60 (Conv2D)              (None, 59, 59, 6)    330         p_gaussian_3[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_4 (PGaussian)        (None, 59, 59, 6)    1           conv2d_60[0][0]                  
  __________________________________________________________________________________________________
  conv2d_61 (Conv2D)              (None, 57, 57, 6)    330         p_gaussian_4[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_5 (PGaussian)        (None, 57, 57, 6)    1           conv2d_61[0][0]                  
  __________________________________________________________________________________________________
  conv2d_62 (Conv2D)              (None, 55, 55, 6)    330         p_gaussian_5[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_6 (PGaussian)        (None, 55, 55, 6)    1           conv2d_62[0][0]                  
  __________________________________________________________________________________________________
  conv2d_63 (Conv2D)              (None, 53, 53, 6)    330         p_gaussian_6[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_7 (PGaussian)        (None, 53, 53, 6)    1           conv2d_63[0][0]                  
  __________________________________________________________________________________________________
  conv2d_64 (Conv2D)              (None, 51, 51, 6)    330         p_gaussian_7[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_8 (PGaussian)        (None, 51, 51, 6)    1           conv2d_64[0][0]                  
  __________________________________________________________________________________________________
  conv2d_65 (Conv2D)              (None, 49, 49, 6)    330         p_gaussian_8[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_9 (PGaussian)        (None, 49, 49, 6)    1           conv2d_65[0][0]                  
  __________________________________________________________________________________________________
  conv2d_66 (Conv2D)              (None, 47, 47, 1)    55          p_gaussian_9[0][0]               
  __________________________________________________________________________________________________
  reshape (Reshape)               (None, 47, 47)       0           conv2d_66[0][0]                  
  __________________________________________________________________________________________________
  flatten (Flatten)               (None, 2209)         0           reshape[0][0]                    
  __________________________________________________________________________________________________
  dense_1 (Dense)                 (None, 2000)         4420000     flatten[0][0]                    
  __________________________________________________________________________________________________
  dense_2 (Dense)                 (None, 200)          442000      flatten[0][0]                    
  __________________________________________________________________________________________________
  softmax (Softmax)               (None, 2000)         0           dense_1[0][0]                    
  __________________________________________________________________________________________________
  softmax_1 (Softmax)             (None, 200)          0           dense_2[0][0]                    
  ==================================================================================================
  Total params: 4,869,857
  Trainable params: 7,857
  Non-trainable params: 4,862,000
  __________________________________________________________________________________________________
  </code></pre>
  
Tags: <convolutional-neural-networks><weights>
Title: Param count in last layer high, how can I decrease?
UserDisplayName: ZeroMaxinumXZ
ThreadText: # Param count in last layer high, how can I decrease?
  
  (Asked by: username_0 on 2020-01-28)
  
  Not sure where to put this... I am trying to create a convolutional architecture for a DQN in keras, and I want to know why my param count is so high for my last layer compared to the rest of the network. I've tried slowly decreasing the dimensions of the layers above it, but it performs quite poorly. I want to know if there's anything I can do to decrease the param count of that last layer, besides the above.
  
  Code:
  
  ```
  #Import statements.
  import random
  import numpy as np
  import tensorflow as tf
  import tensorflow.keras.layers as L
  from collections import deque
  import layers as mL
  import tensorflow.keras.optimizers as O
  import optimizers as mO
  import tensorflow.keras.backend as K
  
  #Conv function.
  def conv(x, units, kernel, stride, noise=False, padding='valid'):
      y = L.Conv2D(units, kernel, stride, activation=mish, padding=padding)(x)
      if noise:
          y = mL.PGaussian()(y)
      return y
  
  #Network
          x_input = L.Input(shape=self.state)
          x_goal = L.Input(shape=self.state)
          x = L.Concatenate(-1)([x_input, x_goal])
          x_list = []
          for i in range(2):
              x = conv(x, 4, (7,7), 1)
          for i in range(2):
              x = conv(x, 8, (5,5), 2)
          for i in range(10):
              x = conv(x, 6, (3,3), 1, noise=True)
          x = L.Conv2D(1, (3,3), 1)(x)
          x_shape = K.int_shape(x)
          x = L.Reshape((x_shape[1], x_shape[2]))(x)
          x = L.Flatten()(x)
          crit = L.Dense(1, trainable=False)(x)
          critic = tf.keras.models.Model([x_input, x_goal], crit)
          act1 = L.Dense(self.action, trainable=False)(x)
          act2 = L.Dense(self.action2, trainable=False)(x)
          act1 = L.Softmax()(act1)
          act2 = L.Softmax()(act2)
          actor = tf.keras.models.Model([x_input, x_goal], [act1, act2])
          actor.compile(loss=mish_loss, optimizer='adam')
          actor.summary()
  
  ```
  
  actor.summary():
  
  ```
  ________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_2 (InputLayer)            [(None, 300, 300, 3) 0                                            
  __________________________________________________________________________________________________
  input_3 (InputLayer)            [(None, 300, 300, 3) 0                                            
  __________________________________________________________________________________________________
  concatenate (Concatenate)       (None, 300, 300, 6)  0           input_2[0][0]                    
                                                                   input_3[0][0]                    
  __________________________________________________________________________________________________
  conv2d_52 (Conv2D)              (None, 294, 294, 4)  1180        concatenate[0][0]                
  __________________________________________________________________________________________________
  conv2d_53 (Conv2D)              (None, 288, 288, 4)  788         conv2d_52[0][0]                  
  __________________________________________________________________________________________________
  conv2d_54 (Conv2D)              (None, 142, 142, 8)  808         conv2d_53[0][0]                  
  __________________________________________________________________________________________________
  conv2d_55 (Conv2D)              (None, 69, 69, 8)    1608        conv2d_54[0][0]                  
  __________________________________________________________________________________________________
  conv2d_56 (Conv2D)              (None, 67, 67, 6)    438         conv2d_55[0][0]                  
  __________________________________________________________________________________________________
  p_gaussian (PGaussian)          (None, 67, 67, 6)    1           conv2d_56[0][0]                  
  __________________________________________________________________________________________________
  conv2d_57 (Conv2D)              (None, 65, 65, 6)    330         p_gaussian[0][0]                 
  __________________________________________________________________________________________________
  p_gaussian_1 (PGaussian)        (None, 65, 65, 6)    1           conv2d_57[0][0]                  
  __________________________________________________________________________________________________
  conv2d_58 (Conv2D)              (None, 63, 63, 6)    330         p_gaussian_1[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_2 (PGaussian)        (None, 63, 63, 6)    1           conv2d_58[0][0]                  
  __________________________________________________________________________________________________
  conv2d_59 (Conv2D)              (None, 61, 61, 6)    330         p_gaussian_2[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_3 (PGaussian)        (None, 61, 61, 6)    1           conv2d_59[0][0]                  
  __________________________________________________________________________________________________
  conv2d_60 (Conv2D)              (None, 59, 59, 6)    330         p_gaussian_3[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_4 (PGaussian)        (None, 59, 59, 6)    1           conv2d_60[0][0]                  
  __________________________________________________________________________________________________
  conv2d_61 (Conv2D)              (None, 57, 57, 6)    330         p_gaussian_4[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_5 (PGaussian)        (None, 57, 57, 6)    1           conv2d_61[0][0]                  
  __________________________________________________________________________________________________
  conv2d_62 (Conv2D)              (None, 55, 55, 6)    330         p_gaussian_5[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_6 (PGaussian)        (None, 55, 55, 6)    1           conv2d_62[0][0]                  
  __________________________________________________________________________________________________
  conv2d_63 (Conv2D)              (None, 53, 53, 6)    330         p_gaussian_6[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_7 (PGaussian)        (None, 53, 53, 6)    1           conv2d_63[0][0]                  
  __________________________________________________________________________________________________
  conv2d_64 (Conv2D)              (None, 51, 51, 6)    330         p_gaussian_7[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_8 (PGaussian)        (None, 51, 51, 6)    1           conv2d_64[0][0]                  
  __________________________________________________________________________________________________
  conv2d_65 (Conv2D)              (None, 49, 49, 6)    330         p_gaussian_8[0][0]               
  __________________________________________________________________________________________________
  p_gaussian_9 (PGaussian)        (None, 49, 49, 6)    1           conv2d_65[0][0]                  
  __________________________________________________________________________________________________
  conv2d_66 (Conv2D)              (None, 47, 47, 1)    55          p_gaussian_9[0][0]               
  __________________________________________________________________________________________________
  reshape (Reshape)               (None, 47, 47)       0           conv2d_66[0][0]                  
  __________________________________________________________________________________________________
  flatten (Flatten)               (None, 2209)         0           reshape[0][0]                    
  __________________________________________________________________________________________________
  dense_1 (Dense)                 (None, 2000)         4420000     flatten[0][0]                    
  __________________________________________________________________________________________________
  dense_2 (Dense)                 (None, 200)          442000      flatten[0][0]                    
  __________________________________________________________________________________________________
  softmax (Softmax)               (None, 2000)         0           dense_1[0][0]                    
  __________________________________________________________________________________________________
  softmax_1 (Softmax)             (None, 200)          0           dense_2[0][0]                    
  ==================================================================================================
  Total params: 4,869,857
  Trainable params: 7,857
  Non-trainable params: 4,862,000
  __________________________________________________________________________________________________
  
  ```
  
  **Tags:** convolutional-neural-networks, weights
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-28. Score: 1 | Accepted answer
  
  If I understood correctly you want to decrease the parameters count on the last layer (`dense_2` layer right?). It would be nice to know why you want to decrease the number of parameters in the last layers... But I'll proceed with what I see.
  
  Firstly, the dense layers (or fully connected in literature) have a deterministic number of parameters (or weights) to learn according to the size of the input and output tensor. The relation is the following:
  
  $N\_{params} = Y\_{output} \cdot (X\_{input} +1) = 200 \cdot (2209 +1) = 442000$
  
  Where:
  
  * $Y\_{output}$: Output tensor shape (`act2=200` which is the output of `dense_2` layer)
  * $X\_{input}$: Input tensor shape (`x=2209`which is the ouput of `flatten` layer)
  
  So if you want to decrease the number of parameters you can:
  
  * Decrease the input tensor: your `self.action2` which I guess is the action space so you might not be able to decrease it
  * Decrease the output tensor: maybe? I would need more context (code) to know if that is even possible
  
  So, in short: if you are not willing to change your the input or output tensor to the Dense layers, then no, you can not decrease the number of parameters.
  
  ---
  
  **BONUS**: In case you missed it, I have noticed you have set your dense layers to `trainable=False`. So in principle you should not care about decreasing number of parameters (which in most cases is motivated to reduce training time) since they are already not being trained. You can check that in the Keras summary output:
  
  ```
  Total params: 4,869,857
  Trainable params: 7,857
  Non-trainable params: 4,862,000
  
  ```
  
  Where the non-trainable parameters are $4862000 = 4420000 + 442000 $, which are the number of parameters of your 2 dense layers.
  
  > **Comments:**
  > **username_0**: If I froze the action layer, would it degrade performance? Likewise, if I bottlenecked the layer above the action layers, wouldn't there be a loss of information?  
  > **username_1**: What is the action layer you are referring to?... I mean, you already froze the 2 dense layers which outputs `act1` and `act2` by setting `trainable=False`. Why did you freeze them? I you posted more code I would be able to get the context of it.  
  
  ---
  
  