Answers.Body: <p>You can:</p>
  <ol>
  <li>Just concatenate the channels of each input image, and then have the usual CNN , or</li>
  <li>Define one feature extractor network (aka backbone), and apply each backbone on each input image such that to <em>share weights</em> among the inputs, then you have the &quot;head&quot; part of the network with the various layers (e.g., classification head, detection head, etc.)</li>
  <li>Alternatively, if the content of each image is quite different you can train <span class="math-container">$N$</span> backbones (so no more weight sharing, and so saving parameters, as in approach 2), followed by the head part.</li>
  </ol>
  <p>Also, in both approach 2 and 3 you need a way to combine the last feature maps of the backbone(s). To do so you can just: <em>sum</em>, <em>multiply</em>, or <em>concatenate</em> them.</p>
  <p>To recap, the architecture in approach 2 or 3 would be: <code>images -&gt; [backbone(img) for img in images] -&gt; combine (e.g., concat) -&gt; output-head.</code></p>
  <p>Indeed, approach 1 is the simplest but also likely to be less performant, whereas the approach 3 is expected to have best accuracy at the cost of increased training cost and slower inference speed. Therefore, approach 2 is the compromise between the two.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-28T13:09:32.070Z
Answers.Text: So solution 2 is like creating an auto encoder trained on all images and then concatenating the latent vectors as input to the head?
Answers.UserDisplayName: Erik Storm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-29T14:07:11.280Z
Answers.Text: @ErikStorm Well, kind of. I prefer to call it a "feature extractor" which yields a feature vector (similar to a large latent space) or low-dimensional feature maps. Yes, then you can concatenate the vectors for the head part of the network.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T21:06:46.237Z
Answers.LastActivityDate: 2023-10-27T21:06:46.237Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I want to train a convolutional neural network on multiple input images. My image is 240x360 and is in RGB. Therefore my input image has a shape of 3x240x360. Now I want to use multiple images of the same size. Do I stack the images and increase the input channel size of my CNN or do I have to create another CNN and then concatenate the tensors later on? I assume both could work but would there be a difference between the two implementations? You can imagine that my problem is like with a self driving Tesla. A Tesla has 8 video streams that all have the same dimensions. The images will probably share the same features e.g. a car or a pedestrian, but in which image it occurs in will make a difference in meaning. What would be the best architecture for this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-30T20:37:52.853Z
Comments.Text: What is your input layer size? If your input layer is 360 x 240 and if you give larger than that, you object in the image would be too small to learn for neural network. Have you consider utilizing GPU more to speed up training?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><convolutional-neural-networks>
Title: How do I train a CNN on multiple images that have the same shape?
UserDisplayName: Erik Storm
ThreadText: # How do I train a CNN on multiple images that have the same shape?
  
  (Asked by: username_0 on 2023-10-27)
  
  I want to train a convolutional neural network on multiple input images. My image is 240x360 and is in RGB. Therefore my input image has a shape of 3x240x360. Now I want to use multiple images of the same size. Do I stack the images and increase the input channel size of my CNN or do I have to create another CNN and then concatenate the tensors later on? I assume both could work but would there be a difference between the two implementations? You can imagine that my problem is like with a self driving Tesla. A Tesla has 8 video streams that all have the same dimensions. The images will probably share the same features e.g. a car or a pedestrian, but in which image it occurs in will make a difference in meaning. What would be the best architecture for this?
  
  **Tags:** neural-networks, convolutional-neural-networks
  > **Comments:**
  > **username_1**: What is your input layer size? If your input layer is 360 x 240 and if you give larger than that, you object in the image would be too small to learn for neural network. Have you consider utilizing GPU more to speed up training?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-27. Score: 2
  
  You can:
  
  1. Just concatenate the channels of each input image, and then have the usual CNN , or
  2. Define one feature extractor network (aka backbone), and apply each backbone on each input image such that to *share weights* among the inputs, then you have the "head" part of the network with the various layers (e.g., classification head, detection head, etc.)
  3. Alternatively, if the content of each image is quite different you can train $N$ backbones (so no more weight sharing, and so saving parameters, as in approach 2), followed by the head part.
  
  Also, in both approach 2 and 3 you need a way to combine the last feature maps of the backbone(s). To do so you can just: *sum*, *multiply*, or *concatenate* them.
  
  To recap, the architecture in approach 2 or 3 would be: `images -> [backbone(img) for img in images] -> combine (e.g., concat) -> output-head.`
  
  Indeed, approach 1 is the simplest but also likely to be less performant, whereas the approach 3 is expected to have best accuracy at the cost of increased training cost and slower inference speed. Therefore, approach 2 is the compromise between the two.
  
  > **Comments:**
  > **username_0**: So solution 2 is like creating an auto encoder trained on all images and then concatenating the latent vectors as input to the head?  
  > **username_2**: @ErikStorm Well, kind of. I prefer to call it a "feature extractor" which yields a feature vector (similar to a large latent space) or low-dimensional feature maps. Yes, then you can concatenate the vectors for the head part of the network.  
  
  ---
  
  