Answers.Body: <p>You can combine the image output using concatenation. Please refer to this paper:</p>
  
  <p><a href="http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf" rel="nofollow noreferrer">http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf</a></p>
  
  <p>You can have a look at the Figure 2. And if you are using caffe, there is a layer called Concat layer. You can use it for your purpose.</p>
  
  <p>I am not fully clear about what you want to do. But like you said, if you want to pass the image values from the first layer to some layers. Try reading about skip architectures.</p>
  
  <p>If you want to use this network as real/fake finder, you can take the difference between two images and convert it to classification problem.</p>
  
  <p>Hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-03T12:45:03.773Z
Answers.Text: Note that the website linked is under construction. You can still find the paper here: https://web.archive.org/web/20170517022842/http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-28T02:05:05.570Z
Answers.LastActivityDate: 2017-06-28T02:05:05.570Z
Answers.UserDisplayName: Dharma
Answers.Body: <p><a href="https://ai.stackexchange.com/a/5187/23994]">eggie5</a> actually has a good solution for you. This approach is a tried and tested way to solve the same problem you are trying to solve.  </p>
  
  <p>However, if you still want to concatenate the images and do this your way, you should <strong>concatenate the images along the channel dimension</strong>.</p>
  
  <p>For example, by combining two <span class="math-container">$200\times 100 \times c$</span> feature vectors (where c is the number of channels) you should get a single <span class="math-container">$200\times 100 \times 2c$</span> feature vector. </p>
  
  <p>The kernels of the next convolution look through all the channels of the feature vector <span class="math-container">$x \times x$</span> pixels at a time.<br>
  If we combine along the channel dimension, it becomes easier for the network to compare pixel values at corresponding positions in both images. Since the objective is to predict similarity or dissimilarity, this is ideal for us.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T18:25:44.727Z
Answers.Text: Hey, welcome to the site. Could you edit your answer to explain *why* concatenation should be along this dimension?
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T15:43:35.970Z
Answers.LastActivityDate: 2019-04-18T23:44:20.160Z
Answers.LastEditDate: 2019-04-18T23:44:20.160Z
Answers.UserDisplayName: Tanmay
Answers.Body: <p>I'm not sure what you mean by pairs. But a common pattern for dealing w/ pair-wise ranking is a siamese network:</p>
  
  <p><a href="https://i.sstatic.net/3zG0B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3zG0B.png" alt="enter image description here"></a></p>
  
  <p>Where A and B are a a pos, negative pair and then the Feature Generation Block is a CNN architecture which outputs a feature vector for each image (cut off the softmax) and then the network tried to maximise the regression loss between the two images. The two networks share the same parameters and thus in the end you have one model which can accurately disambiguate between a positive or negative pair.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-29T09:48:47.500Z
Answers.LastActivityDate: 2018-01-29T09:48:47.500Z
Answers.UserDisplayName: eggie5
Body: <p>For a classification task (I'm showing a pair of exactly two images to a CNN that should answer with 0 -> fake pair or 1 -> real pair) I am struggling to figure out how to design the input.</p>
  
  <p>At the moment the network's architecture looks like this:</p>
  
  <pre><code>image-1                       image-2
     |                             |
  conv layer                    conv layer
     |                             |
     _______________ _______________
                    |
              flattened vector
                    |
            fully-connected layer
                    |
             reshape to 2D image
                    |
                conv layer
                    |
                conv layer
                    |
                conv layer
                    |
              flattened vector
                    |
                  output
  </code></pre>
  
  <p>The conv layers have a <code>2x2</code> stride, thus halfing the images' dimensions. I would have used the first fully-connected layer as the first layer, but then the size of it doesn't fit in my GPU's VRAM. Thus, I have the first conv layers halfing the size of the images first, then combining the information with a fully-connected layer and then doing the actual classification with conv layers for the combined image information.</p>
  
  <p>My very first idea was to simply add the information up, like <code>(image-1 + image-2) / 2</code>...but this is not a good idea, since it heavily mixes up image information.</p>
  
  <p>The next try was to concatenate the images to have one single image of size 400x100 instead of two 200x100 images. However, the results of this approach were quite unstable. I think because in the center of the big, concatenated image convolutions would convolve information of both images (right border of <code>image-1</code> / left border of <code>image-2</code>), which again mixes up image information in not really senseful way.</p>
  
  <p>My last approach was the current architecture, simply leaving the combination of <code>image-1</code> and <code>image-2</code> up to one fully-connected layer. This works - kind of (the results show a nice convergence, but could be better).</p>
  
  <p>What is a reasonable, "state-of-the-art" way to combine two images for a CNN's input?</p>
  
  <p>I clearly can not simply increase the batch size and fit the images there, since the pairs are related to each other and this relationship would get lost if I simply feed just one image at a time and increase the batch size.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-09T09:18:36.783Z
Comments.Text: Could you give more information on what makes a "real pair" and a "fake pair"?
Comments.UserDisplayName: Thomas Wagenaar
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-09T09:20:41.227Z
Comments.Text: I've implemented a GAN. Should've mentioned that earlier. Fake pair is a generated sample, real pair is training images.
Comments.UserDisplayName: daniel451
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-09T09:24:52.670Z
Comments.Text: To be specific, I'm trying to implement my own version of this: https://phillipi.github.io/pix2pix/
Comments.UserDisplayName: daniel451
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-09T09:46:23.683Z
Comments.Text: the only thing I could think of is passing both images through a whole set of convolutional layers and pooling layers on their own (independently), and then combining them with a fully connected layer at the end of the network.
Comments.UserDisplayName: Thomas Wagenaar
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-16T10:15:02.810Z
Comments.Text: @ThomasW I have only tested one setup so far with two completely different sets/streams of conv layers for the two images and the "late fusion", but this one test did perform worse.
Comments.UserDisplayName: daniel451
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-16T10:17:52.007Z
Comments.Text: My latest approach is to simply concat the feature maps after the first separated conv layers, i.e. concatenating 32 feature maps for `image-1` with 32 feature maps for `image-2` to get one large set of 64 feature maps and then pass them to the subsequent layers. Still rather unstable, but best performance so far.
Comments.UserDisplayName: daniel451
Tags: <neural-networks><machine-learning><convolutional-neural-networks><image-recognition><classification>
Title: How to "combine" two images for CNN input (classification task)?
UserDisplayName: daniel451
ThreadText: # How to "combine" two images for CNN input (classification task)?
  
  (Asked by: username_0 on 2017-05-08)
  
  For a classification task (I'm showing a pair of exactly two images to a CNN that should answer with 0 -> fake pair or 1 -> real pair) I am struggling to figure out how to design the input.
  At the moment the network's architecture looks like this:
  image-1                       image-2
     |                             |
  conv layer                    conv layer
     |                             |
     _______________ _______________
                    |
              flattened vector
                    |
            fully-connected layer
                    |
             reshape to 2D image
                    |
                conv layer
                    |
                conv layer
                    |
                conv layer
                    |
              flattened vector
                    |
                  output
  
  The conv layers have a 2x2 stride, thus halfing the images' dimensions. I would have used the first fully-connected layer as the first layer, but then the size of it doesn't fit in my GPU's VRAM. Thus, I have the first conv layers halfing the size of the images first, then combining the information with a fully-connected layer and then doing the actual classification with conv layers for the combined image information.
  My very first idea was to simply add the information up, like (image-1 + image-2) / 2...but this is not a good idea, since it heavily mixes up image information.
  The next try was to concatenate the images to have one single image of size 400x100 instead of two 200x100 images. However, the results of this approach were quite unstable. I think because in the center of the big, concatenated image convolutions would convolve information of both images (right border of image-1 / left border of image-2), which again mixes up image information in not really senseful way.
  My last approach was the current architecture, simply leaving the combination of image-1 and image-2 up to one fully-connected layer. This works - kind of (the results show a nice convergence, but could be better).
  What is a reasonable, "state-of-the-art" way to combine two images for a CNN's input?
  I clearly can not simply increase the batch size and fit the images there, since the pairs are related to each other and this relationship would get lost if I simply feed just one image at a time and increase the batch size.
  
  **Tags:** neural-networks, machine-learning, convolutional-neural-networks, image-recognition, classification
  > **Comments:**
  > **username_1**: Could you give more information on what makes a "real pair" and a "fake pair"?  
  > **username_0**: I've implemented a GAN. Should've mentioned that earlier. Fake pair is a generated sample, real pair is training images.  
  > **username_0**: To be specific, I'm trying to implement my own version of this: https://phillipi.github.io/pix2pix/  
  > **username_1**: the only thing I could think of is passing both images through a whole set of convolutional layers and pooling layers on their own (independently), and then combining them with a fully connected layer at the end of the network.  
  > **username_0**: @ThomasW I have only tested one setup so far with two completely different sets/streams of conv layers for the two images and the "late fusion", but this one test did perform worse.  
  > **username_0**: My latest approach is to simply concat the feature maps after the first separated conv layers, i.e. concatenating 32 feature maps for `image-1` with 32 feature maps for `image-2` to get one large set of 64 feature maps and then pass them to the subsequent layers. Still rather unstable, but best performance so far.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2017-06-28. Score: 2 | Accepted answer
  
  You can combine the image output using concatenation. Please refer to this paper:
  
  <http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf>
  
  You can have a look at the Figure 2. And if you are using caffe, there is a layer called Concat layer. You can use it for your purpose.
  
  I am not fully clear about what you want to do. But like you said, if you want to pass the image values from the first layer to some layers. Try reading about skip architectures.
  
  If you want to use this network as real/fake finder, you can take the difference between two images and convert it to classification problem.
  
  Hope it helps.
  
  > **Comments:**
  > **username_3**: Note that the website linked is under construction. You can still find the paper here: https://web.archive.org/web/20170517022842/http://ivpl.eecs.northwestern.edu/sites/default/files/07444187.pdf  
  
  ---
  
  ### Answer by username_4 on 2019-04-17. Score: 1
  
  [eggie5](https://ai.stackexchange.com/a/5187/23994]) actually has a good solution for you. This approach is a tried and tested way to solve the same problem you are trying to solve.
  
  However, if you still want to concatenate the images and do this your way, you should **concatenate the images along the channel dimension**.
  
  For example, by combining two $200\times 100 \times c$ feature vectors (where c is the number of channels) you should get a single $200\times 100 \times 2c$ feature vector.
  
  The kernels of the next convolution look through all the channels of the feature vector $x \times x$ pixels at a time.  
  If we combine along the channel dimension, it becomes easier for the network to compare pixel values at corresponding positions in both images. Since the objective is to predict similarity or dissimilarity, this is ideal for us.
  
  > **Comments:**
  > **username_5**: Hey, welcome to the site. Could you edit your answer to explain *why* concatenation should be along this dimension?  
  
  ---
  
  ### Answer by username_6 on 2018-01-29. Score: 1
  
  I'm not sure what you mean by pairs. But a common pattern for dealing w/ pair-wise ranking is a siamese network:
  
  [![enter image description here](https://i.sstatic.net/3zG0B.png)](https://i.sstatic.net/3zG0B.png)
  
  Where A and B are a a pos, negative pair and then the Feature Generation Block is a CNN architecture which outputs a feature vector for each image (cut off the softmax) and then the network tried to maximise the regression loss between the two images. The two networks share the same parameters and thus in the end you have one model which can accurately disambiguate between a positive or negative pair.
  
  
  ---
  
  