Answers.Body: <p>An interesting issue: the model is seemingly under-fitting to the train data (since the accuracy isn't that high), but it is also underfitting to the training data. :)</p>
  <p>When you flatten the data and use dense layers, you lose all of the spatial information. Before you do that, you might want to subtract the left side of the output (I calculated it having a size of 15x30x128, so a half would be 15x15x128?) from the right side. This is a bit tricky with the <code>Sequential</code> model, but it can be done with a <code>Lambda</code> layer. Note that the tensor's first dimension is <code>None</code>, which is actually the batch size during training.</p>
  <p>After doing this you may keep the current flatten + dense structure and see how it goes. Although I suppose it can be much simpler at this point, Conv2d is doing the heavy lifting.</p>
  <p>You might want to add dropouts to the earlier part of the model as well, or use enough data augmentation to avoid overfitting. The black bar in the middle is obstructing the image, making them very non-identical.</p>
  <p>Alternatively you could make a single feature extraction model and feed it these two inputs separately. This architecture is called a <a href="https://towardsdatascience.com/a-friendly-introduction-to-siamese-networks-85ab17522942" rel="nofollow noreferrer">Siamese Network</a>, and requires a bit more code to implement (example in the article, however it uses PyTorch which seems quite verbose when compared to Keras).</p>
  <p>Actually the first suggestion is a Siamese Network as well, just in a bit odd structure having just one input. I'd be interested to hear if this works better for you.</p>
  <p><strong>Edit:</strong> I decided to try out the problem myself. Using the idea described above, I got 99% validation accuracy on images which aren't even identical, using a model with 194 parameters. Half of the examples have a different image, and the other half is a modified copy of the reference (random +/- 10 degrees of rotation, +/- 10% of zoom with TF's <code>ImageDataGenerator</code>).</p>
  <p>I won't share the code yet in case you want to figure it out yourself, but here is the summary which shows the dataflow (uses the <code>Sequential</code> API):</p>
  <pre><code>Model: &quot;sequential_173&quot;
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv2d_132 (Conv2D)          (None, 126, 254, 3)       84        
  _________________________________________________________________
  max_pooling2d_88 (MaxPooling (None, 63, 127, 3)        0         
  _________________________________________________________________
  conv2d_133 (Conv2D)          (None, 61, 125, 3)        84        
  _________________________________________________________________
  max_pooling2d_89 (MaxPooling (None, 30, 62, 3)         0         
  _________________________________________________________________
  lambda_60 (Lambda)           (None, 30, 30, 3)         0         
  _________________________________________________________________
  average_pooling2d_47 (Averag (None, 5, 5, 3)           0         
  _________________________________________________________________
  lambda_61 (Lambda)           (None, 5, 5)              0         
  _________________________________________________________________
  flatten_52 (Flatten)         (None, 25)                0         
  _________________________________________________________________
  dense_435 (Dense)            (None, 1)                 26        
  =================================================================
  Total params: 194
  Trainable params: 194
  Non-trainable params: 0
  _________________________________________________________________
  </code></pre>
  <p>It takes a &quot;twin&quot; RGB image in, and applies two <code>Conv2D</code> layers with just 3 kernels on each of them. This leads to <code>30 x 31 x 3</code> shaped image features (the width of <code>max_pooling2d_89</code>'s output is actually <code>62</code> since there are two images side-by-side).</p>
  <p>The two sides are subtracted from each other (cropping out the middle column) and the result is downsampled and aggregated, resulting in a <code>5 x 5</code> output. The final <code>Dense</code> layers is just a <code>sigmoid</code> to predict whether the images are nearly identical or not.</p>
  <p>Example result is shown below. It has six columns of images: left image, right image, left's features (<code>conv2d_133</code>'s output), right's features, feature difference and weighted feature difference. The last column shows that the model has learned to focus more on the center of the image rather than corners, because the used image augmentation distorts the corners the most.</p>
  <p><a href="https://i.sstatic.net/KHk2h.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KHk2h.jpg" alt="Result example" /></a></p>
  <p>If the image is downsampled to just <code>1 x 1</code> before the <code>Flatten</code> layer then the model has only 170 parameters, but the validation accuracy drops a bit to 97%. (<strong>edit 2</strong>: using <code>elu</code> rather than <code>tanh</code> activation brings this model's accuracy to 99.3%, I originally used <code>tanh</code> since it is easy to visualize as an image due to its limited range)</p>
  <p>The used dataset is <a href="http://web.mit.edu/torralba/www/indoor.html" rel="nofollow noreferrer">Indoor Scene Recognition</a>, of which I used 2000 random images.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T20:43:10.830Z
Answers.LastActivityDate: 2022-01-07T16:36:03.330Z
Answers.LastEditDate: 2022-01-07T16:36:03.330Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I have been running some experiments to see whether a CNN can detect whether two images are the same. However, I can't seem to make it work. I am wondering whether CNNs are not able to do what I am trying, or whether I am making mistakes.</p>
  <p>This is the setup: I take two 128x128 images and append them to be 128x256. These two images are either exactly the same:
  <a href="https://i.sstatic.net/qJLVp.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qJLVp.jpg" alt="two images of class 'same'" /></a>
  or different:
  <a href="https://i.sstatic.net/f6n0t.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/f6n0t.jpg" alt="two images of class 'not_same'" /></a></p>
  <p>I have also experimented with a black bar in the middle. The entire appended 128x256 image(thus containing two images) is fed into a CNN with the following setup:
  <a href="https://i.sstatic.net/mrST1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mrST1.png" alt="enter image description here" /></a></p>
  <p>Depending on the dropout usage, the model is able to learn training data between 80-100% accuracy. But the validation accuracy never gets above around 65%.</p>
  <p>I do not understand why the model is not able to get to 100% very easily, as the input images are either EXACTLY the same, or very different. Seems like a trivial task to me. I would assume that the convolutional kernels would create exactly the same feature representations of the images and then feed that into the dense layers.</p>
  <p>ps. I know there are better ways to do this but I wanted to experiment with this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T15:33:49.900Z
Comments.Text: As you state, the main problem is why in the training you can get 80-100% but not in the test, so, in this case a lot of factors may affect this, and we need more information
Comments.UserDisplayName: malioboro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T15:35:41.443Z
Comments.Text: For example, What is your test data like? how many data train do you have? how about the ratio? do you use the same object/distribution? how do you validate? do you use cross validation? (these questions check if there is overfitting)
Comments.UserDisplayName: malioboro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T15:37:15.860Z
Comments.Text: "why the model is not able to get to 100% very easily" even in simple image classification with small classess (not similarity), it's not easy to get 100% in validation
Comments.UserDisplayName: malioboro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T20:56:18.510Z
Comments.Text: Can you please provide the actual code or link to code rather than a screenshot of the code? In reality, almost no question asked on this site should require users to show us code, but sometimes it can be useful. I would like to note that programming questions are off-topic here and I would recommend that you read our on-topic page. https://ai.stackexchange.com/help/on-topic
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T06:08:54.573Z
Comments.Text: This is a wild guess, but I have a strong feeling that this problem can be easily solved by using a deeper neural network. Try using well-known architectures like Resnet first and see if you still have the same issue. If not, the model is underfitting due to its lack of complexity.
Comments.UserDisplayName: DKDK
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-07T14:35:34.760Z
Comments.Text: @DKDK I got 97% accuracy with just 170 model parameters on a more difficult dataset ("identical" images are actually augmented), see my answer below.
Comments.UserDisplayName: NikoNyrh
Tags: <convolutional-neural-networks><similarity>
Title: Can CNNs detect image similarity?
UserDisplayName: Bart
ThreadText: # Can CNNs detect image similarity?
  
  (Asked by: username_0 on 2022-01-05)
  
  I have been running some experiments to see whether a CNN can detect whether two images are the same. However, I can't seem to make it work. I am wondering whether CNNs are not able to do what I am trying, or whether I am making mistakes.
  
  This is the setup: I take two 128x128 images and append them to be 128x256. These two images are either exactly the same:
  [![two images of class 'same'](https://i.sstatic.net/qJLVp.jpg)](https://i.sstatic.net/qJLVp.jpg)
  or different:
  [![two images of class 'not_same'](https://i.sstatic.net/f6n0t.jpg)](https://i.sstatic.net/f6n0t.jpg)
  
  I have also experimented with a black bar in the middle. The entire appended 128x256 image(thus containing two images) is fed into a CNN with the following setup:
  [![enter image description here](https://i.sstatic.net/mrST1.png)](https://i.sstatic.net/mrST1.png)
  
  Depending on the dropout usage, the model is able to learn training data between 80-100% accuracy. But the validation accuracy never gets above around 65%.
  
  I do not understand why the model is not able to get to 100% very easily, as the input images are either EXACTLY the same, or very different. Seems like a trivial task to me. I would assume that the convolutional kernels would create exactly the same feature representations of the images and then feed that into the dense layers.
  
  ps. I know there are better ways to do this but I wanted to experiment with this.
  
  **Tags:** convolutional-neural-networks, similarity
  
  > **Comments:**
  > **username_1**: As you state, the main problem is why in the training you can get 80-100% but not in the test, so, in this case a lot of factors may affect this, and we need more information  
  > **username_1**: For example, What is your test data like? how many data train do you have? how about the ratio? do you use the same object/distribution? how do you validate? do you use cross validation? (these questions check if there is overfitting)  
  > **username_1**: "why the model is not able to get to 100% very easily" even in simple image classification with small classess (not similarity), it's not easy to get 100% in validation  
  > **username_2**: Can you please provide the actual code or link to code rather than a screenshot of the code? In reality, almost no question asked on this site should require users to show us code, but sometimes it can be useful. I would like to note that programming questions are off-topic here and I would recommend that you read our on-topic page. https://ai.stackexchange.com/help/on-topic  
  > **username_3**: This is a wild guess, but I have a strong feeling that this problem can be easily solved by using a deeper neural network. Try using well-known architectures like Resnet first and see if you still have the same issue. If not, the model is underfitting due to its lack of complexity.  
  > **username_4**: @username_3 I got 97% accuracy with just 170 model parameters on a more difficult dataset ("identical" images are actually augmented), see my answer below.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2022-01-05. Score: 1
  
  An interesting issue: the model is seemingly under-fitting to the train data (since the accuracy isn't that high), but it is also underfitting to the training data. :)
  
  When you flatten the data and use dense layers, you lose all of the spatial information. Before you do that, you might want to subtract the left side of the output (I calculated it having a size of 15x30x128, so a half would be 15x15x128?) from the right side. This is a bit tricky with the `Sequential` model, but it can be done with a `Lambda` layer. Note that the tensor's first dimension is `None`, which is actually the batch size during training.
  
  After doing this you may keep the current flatten + dense structure and see how it goes. Although I suppose it can be much simpler at this point, Conv2d is doing the heavy lifting.
  
  You might want to add dropouts to the earlier part of the model as well, or use enough data augmentation to avoid overfitting. The black bar in the middle is obstructing the image, making them very non-identical.
  
  Alternatively you could make a single feature extraction model and feed it these two inputs separately. This architecture is called a [Siamese Network](https://towardsdatascience.com/a-friendly-introduction-to-siamese-networks-85ab17522942), and requires a bit more code to implement (example in the article, however it uses PyTorch which seems quite verbose when compared to Keras).
  
  Actually the first suggestion is a Siamese Network as well, just in a bit odd structure having just one input. I'd be interested to hear if this works better for you.
  
  **Edit:** I decided to try out the problem myself. Using the idea described above, I got 99% validation accuracy on images which aren't even identical, using a model with 194 parameters. Half of the examples have a different image, and the other half is a modified copy of the reference (random +/- 10 degrees of rotation, +/- 10% of zoom with TF's `ImageDataGenerator`).
  
  I won't share the code yet in case you want to figure it out yourself, but here is the summary which shows the dataflow (uses the `Sequential` API):
  
  ```
  Model: "sequential_173"
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv2d_132 (Conv2D)          (None, 126, 254, 3)       84        
  _________________________________________________________________
  max_pooling2d_88 (MaxPooling (None, 63, 127, 3)        0         
  _________________________________________________________________
  conv2d_133 (Conv2D)          (None, 61, 125, 3)        84        
  _________________________________________________________________
  max_pooling2d_89 (MaxPooling (None, 30, 62, 3)         0         
  _________________________________________________________________
  lambda_60 (Lambda)           (None, 30, 30, 3)         0         
  _________________________________________________________________
  average_pooling2d_47 (Averag (None, 5, 5, 3)           0         
  _________________________________________________________________
  lambda_61 (Lambda)           (None, 5, 5)              0         
  _________________________________________________________________
  flatten_52 (Flatten)         (None, 25)                0         
  _________________________________________________________________
  dense_435 (Dense)            (None, 1)                 26        
  =================================================================
  Total params: 194
  Trainable params: 194
  Non-trainable params: 0
  _________________________________________________________________
  
  ```
  
  It takes a "twin" RGB image in, and applies two `Conv2D` layers with just 3 kernels on each of them. This leads to `30 x 31 x 3` shaped image features (the width of `max_pooling2d_89`'s output is actually `62` since there are two images side-by-side).
  
  The two sides are subtracted from each other (cropping out the middle column) and the result is downsampled and aggregated, resulting in a `5 x 5` output. The final `Dense` layers is just a `sigmoid` to predict whether the images are nearly identical or not.
  
  Example result is shown below. It has six columns of images: left image, right image, left's features (`conv2d_133`'s output), right's features, feature difference and weighted feature difference. The last column shows that the model has learned to focus more on the center of the image rather than corners, because the used image augmentation distorts the corners the most.
  
  [![Result example](https://i.sstatic.net/KHk2h.jpg)](https://i.sstatic.net/KHk2h.jpg)
  
  If the image is downsampled to just `1 x 1` before the `Flatten` layer then the model has only 170 parameters, but the validation accuracy drops a bit to 97%. (**edit 2**: using `elu` rather than `tanh` activation brings this model's accuracy to 99.3%, I originally used `tanh` since it is easy to visualize as an image due to its limited range)
  
  The used dataset is [Indoor Scene Recognition](http://web.mit.edu/torralba/www/indoor.html), of which I used 2000 random images.
  
  
  ---
  
  