Answers.Body: <p>A bit late to the party but the only thing I could come up with is VGG-oriented architectures. What I mean by that is use blocks of convolutional layers that leave the input the same by for instance using stride one and same padding. Finish the previous block with a convolutional layer that has stride 2 or valid padding and then create a new block that utilizes a zero padding layer.</p>
  <p>For tensorflow you would do something like this:</p>
  <pre><code>   inputs = keras.Input(shape=input_shape)
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding=&quot;same&quot;, strides=1,activation='relu')(inputs)
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding=&quot;same&quot;, strides=1,activation='relu')(x)
     ...
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding=&quot;same&quot;, strides=2,activation='relu')(x)
     # new block
     x = layers.ZeroPadding2D(padding=((padding_num)(x)
     ...
  </code></pre>
  <p>This way you can have several layers that leave the input size untouched until you want to go further. Ofcourse instead of changing the stride, you could add a MaxPooling layer etc.
  My answer is based on subjective empirical evidence with small images and with trying to expand networks without reducing the size of the initial input to too small dimensions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-19T13:00:12.763Z
Answers.Text: Upvote as this suggestion is helpful. I don't think it should be "the answer" as it seems either the answer is "no, there is no best practice" or at least a consensus approach hasn't arisen. That said, I used pretty much this approach ultimately!
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-15T21:27:45.070Z
Answers.LastActivityDate: 2022-12-15T21:28:02.777Z
Answers.LastEditDate: 2022-12-15T21:28:02.777Z
Answers.UserDisplayName: Nikos H.
Body: <p>With all the work being done on larger and larger images, I'd like to ask if a best practice(s) has arisen for allowing multiple convolutional layers on small image inputs?</p>
  <p>For instance, in my case I have a 7x7x4 input that I convolve (coming from a small visual piece of a RL environment). Obviously if I want to allow a few layers with increasing depth/channels to allow the network to learn abstractions, many kernel/padding combinations will swiftly reduce the first two dimensions to a point where arguably convolutional layers don't make sense.</p>
  <p>For my own case, I've managed to get 3 seemingly reasonable layers by playing with kernel size, stride, and padding, and while it &quot;works&quot; for my use-case, it feels outright arbitrary:</p>
  <pre><code># Values for the first conv layer
  self.ksize_1 = 3
  self.stride_1 = 1
  self.padding_1 = 1
  # retains the input_dimz x input_dimz shape
  
  # Values for the second conv layer
  self.ksize_2 = 4
  self.stride_2 = 1
  self.padding_2 = 1
  # roughly makes dimension even (looks like input_dimz - 1)
  
  # Values for the third conv layer
  self.ksize_3 = 3
  self.stride_3 = 1
  self.padding_3 = 0
  # outputs dimensions roughly 2 less in width by height from above
  # should be (input_dimz - 3) x (input_dimz - 3)
  
  # Now our actual conv layers
  self.conv1 = nn.Conv2d(self.input_chan, self.conv_dim, self.ksize_1,
                         self.stride_1, self.padding_1, padding_mode=self.padding_mode)
  self.conv2 = nn.Conv2d(self.conv_dim, self.conv_dim * 2, self.ksize_2,
                         self.stride_2, self.padding_2, padding_mode=self.padding_mode)
  self.conv3 = nn.Conv2d(self.conv_dim * 2, self.conv_dim * 3, self.ksize_3,
                         self.stride_3, self.padding_3, padding_mode=self.padding_mode)
  </code></pre>
  <p>(where <code>conv_dim</code> is essentially a hyperparam to tweak the channel depth)</p>
  <p>Are small dimensional cases like this still pretty arbitrary and the above reasonable, or has a methodological approach arisen that can be followed?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T01:44:27.643Z
Comments.Text: Hi @Josh, are you saying that you have 4 "channel dimensions"?  Is it RGB + alpha?  Also, are your images really only 7x7-pixels?
Comments.UserDisplayName: Snehal Patel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-26T12:34:49.057Z
Comments.Text: @SnehalPatel yep 4 channel dimensions and indeed 7x7. In my case it's visual-like info coming from a piece of a reinforcement learning environment
Comments.UserDisplayName: Josh
Tags: <convolutional-neural-networks><computer-vision><convolution><convolutional-layers>
Title: Is there a best practice for creating multiple convolutional layers from small image inputs?
UserDisplayName: Josh
ThreadText: # Is there a best practice for creating multiple convolutional layers from small image inputs?
  
  (Asked by: username_0 on 2022-10-04)
  
  With all the work being done on larger and larger images, I'd like to ask if a best practice(s) has arisen for allowing multiple convolutional layers on small image inputs?
  For instance, in my case I have a 7x7x4 input that I convolve (coming from a small visual piece of a RL environment). Obviously if I want to allow a few layers with increasing depth/channels to allow the network to learn abstractions, many kernel/padding combinations will swiftly reduce the first two dimensions to a point where arguably convolutional layers don't make sense.
  For my own case, I've managed to get 3 seemingly reasonable layers by playing with kernel size, stride, and padding, and while it "works" for my use-case, it feels outright arbitrary:
  # Values for the first conv layer
  self.ksize_1 = 3
  self.stride_1 = 1
  self.padding_1 = 1
  # retains the input_dimz x input_dimz shape
  
  # Values for the second conv layer
  self.ksize_2 = 4
  self.stride_2 = 1
  self.padding_2 = 1
  # roughly makes dimension even (looks like input_dimz - 1)
  
  # Values for the third conv layer
  self.ksize_3 = 3
  self.stride_3 = 1
  self.padding_3 = 0
  # outputs dimensions roughly 2 less in width by height from above
  # should be (input_dimz - 3) x (input_dimz - 3)
  
  # Now our actual conv layers
  self.conv1 = nn.Conv2d(self.input_chan, self.conv_dim, self.ksize_1,
                         self.stride_1, self.padding_1, padding_mode=self.padding_mode)
  self.conv2 = nn.Conv2d(self.conv_dim, self.conv_dim * 2, self.ksize_2,
                         self.stride_2, self.padding_2, padding_mode=self.padding_mode)
  self.conv3 = nn.Conv2d(self.conv_dim * 2, self.conv_dim * 3, self.ksize_3,
                         self.stride_3, self.padding_3, padding_mode=self.padding_mode)
  
  (where conv_dim is essentially a hyperparam to tweak the channel depth)
  Are small dimensional cases like this still pretty arbitrary and the above reasonable, or has a methodological approach arisen that can be followed?
  
  **Tags:** convolutional-neural-networks, computer-vision, convolution, convolutional-layers
  > **Comments:**
  > **username_1**: Hi @Josh, are you saying that you have 4 "channel dimensions"?  Is it RGB + alpha?  Also, are your images really only 7x7-pixels?  
  > **username_0**: @SnehalPatel yep 4 channel dimensions and indeed 7x7. In my case it's visual-like info coming from a piece of a reinforcement learning environment  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-15. Score: 1
  
  A bit late to the party but the only thing I could come up with is VGG-oriented architectures. What I mean by that is use blocks of convolutional layers that leave the input the same by for instance using stride one and same padding. Finish the previous block with a convolutional layer that has stride 2 or valid padding and then create a new block that utilizes a zero padding layer.
  For tensorflow you would do something like this:
     inputs = keras.Input(shape=input_shape)
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=1,activation='relu')(inputs)
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=1,activation='relu')(x)
     ...
     x = layers.Conv2D(filters=num_filters, kernel_size=kern_size, padding="same", strides=2,activation='relu')(x)
     # new block
     x = layers.ZeroPadding2D(padding=((padding_num)(x)
     ...
  
  This way you can have several layers that leave the input size untouched until you want to go further. Ofcourse instead of changing the stride, you could add a MaxPooling layer etc.
  My answer is based on subjective empirical evidence with small images and with trying to expand networks without reducing the size of the initial input to too small dimensions.
  
  > **Comments:**
  > **username_0**: Upvote as this suggestion is helpful. I don't think it should be "the answer" as it seems either the answer is "no, there is no best practice" or at least a consensus approach hasn't arisen. That said, I used pretty much this approach ultimately!  
  
  ---
  
  