Answers.Body: <p>I think the second approach will be the best because it only requires that your training set is annotated with four labels for each of the four corners of the paper sheet.</p>
  <p>This is sort of the idea of a Region Proposal Network which is used in <a href="https://arxiv.org/pdf/1506.01497.pdf" rel="noreferrer">Faster R-CNN</a> (section 3.1).</p>
  <p><a href="https://github.com/pytorch/vision/blob/5e56575e688a85a3bc9dc3c97934dd864b65ce47/torchvision/models/detection/rpn.py#L88-L367" rel="noreferrer">Here</a> is a reference implementation of a Region Proposal Network in PyTorch from the <a href="https://github.com/pytorch/vision/" rel="noreferrer">torchvision</a> library. Notice how the network outputs <code>boxes</code> (in the <code>forward()</code> method) which is a tuple <code>(x1, y1, x2, y2)</code>. From these four coordinates, you could crop the image to the desired paper sheet region.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T16:19:44.387Z
Answers.LastActivityDate: 2022-03-01T16:19:44.387Z
Answers.UserDisplayName: Lars
Answers.Body: <p>You could try U-Net for approach 1.</p>
  <p>This is called the image-to-image translation problem in machine learning. You could see more architectures in this paper:
  <a href="https://arxiv.org/pdf/2101.08629.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2101.08629.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-02T02:24:00.427Z
Answers.Text: A U-net wouldn't help a whole lot if it's being spatially transformed, since the skip connections wouldn't line up correctly between the input and output. That said, I guess a U-net is probably better than an autoencoder-like model without _any_ skip connections, so idk.
Answers.UserDisplayName: The Guy with The Hat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-03T08:56:43.443Z
Answers.Text: @TheGuywithTheHat What kind of neural network would you use as a general tool for the general problem of image=in image=out, and you want the NN the learn transformations of images on thousands of examples? I would be interested in an answer about this, if you have one! Thanks
Answers.UserDisplayName: logijaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-03T22:20:19.943Z
Answers.Text: @logijaz Accurate spatial transformations are something that current neural net architectures are just fundamentally unsuited to solve. U-nets connect pixels that are in the same location in the input and output, but with spatial transformations those pixels don't correspond to one another, so it's pointless. Other architectures can try to transform the image, but will lose a significant amount of detail (e.g. any text will become gibberish or disappear entirely). If you _really_ want to involve a neural net, use it to find the corners, and then use traditional methods to transform the image.
Answers.UserDisplayName: The Guy with The Hat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T15:44:37.503Z
Answers.LastActivityDate: 2022-03-01T15:44:37.503Z
Answers.UserDisplayName: Mehmet Alican Noyan
Body: <p>With an RGB image of a paper sheet with text, I want to obtain an output image which is cropped and deskewed. Example of input:</p>
  <p><a href="https://i.sstatic.net/l64Kn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/l64Kn.png" alt="enter image description here" /></a></p>
  <p>I have tried non-AI tools (such as <code>openCV.findContours</code>) to find the 4 corners of the sheet, but it's not very robust in some lighting conditions, or if there are other elements on the photo.</p>
  <p>So I see two options:</p>
  <ul>
  <li><p>a NN with <code>input=image, output=image</code>, that does <strong>everything</strong> (including the deskewing, and even also the brightness adjustment). I'll just train it with thousands of images.</p>
  </li>
  <li><p>a NN with <code>input=image, output=coordinates_of_4_corners</code>. Then I'll do the cropping + deskewing with a homographic transform, and brightness adjustment with standard non-AI tools</p>
  </li>
  </ul>
  <p>Which approach would you use?</p>
  <p><strong>More generally what kind of architecture of neural network would you use in the general case <code>input=image, output=image</code>?</strong></p>
  <p>Is approach #2, for which input=image, output=coordinates possible? Or is there another segmentation method you would use here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-01T19:12:02.367Z
Comments.Text: have you tried preprocessing the image before applying findcontours? Like contrast boosting, conversion to hsv space, masking with edge detection. A neural net for such task sounds a real overkill to me, and complicate cause of the annotations that it would require.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T02:32:01.360Z
Comments.Text: I agree that a neural net seems like overkill. That being said, this paper seems interesting: "Convolutional Neural Network Architecture for Geometric Matching" Ignacio Rocco, Relja Arandjelovic, Josef Sivic https://openaccess.thecvf.com/content_cvpr_2017/html/Rocco_Convolutional_Neural_Network_CVPR_2017_paper.html
Comments.UserDisplayName: The Guy with The Hat
Tags: <neural-networks><convolutional-neural-networks><image-processing><image-generation>
Title: Image-in image-out neural network architectures
UserDisplayName: logijaz
ThreadText: # Image-in image-out neural network architectures
  
  (Asked by: username_0 on 2022-03-01)
  
  With an RGB image of a paper sheet with text, I want to obtain an output image which is cropped and deskewed. Example of input:
  
  [![enter image description here](https://i.sstatic.net/l64Kn.png)](https://i.sstatic.net/l64Kn.png)
  
  I have tried non-AI tools (such as `openCV.findContours`) to find the 4 corners of the sheet, but it's not very robust in some lighting conditions, or if there are other elements on the photo.
  
  So I see two options:
  
  * a NN with `input=image, output=image`, that does **everything** (including the deskewing, and even also the brightness adjustment). I'll just train it with thousands of images.
  * a NN with `input=image, output=coordinates_of_4_corners`. Then I'll do the cropping + deskewing with a homographic transform, and brightness adjustment with standard non-AI tools
  
  Which approach would you use?
  
  **More generally what kind of architecture of neural network would you use in the general case `input=image, output=image`?**
  
  Is approach #2, for which input=image, output=coordinates possible? Or is there another segmentation method you would use here?
  
  **Tags:** neural-networks, convolutional-neural-networks, image-processing, image-generation
  
  > **Comments:**
  > **username_1**: have you tried preprocessing the image before applying findcontours? Like contrast boosting, conversion to hsv space, masking with edge detection. A neural net for such task sounds a real overkill to me, and complicate cause of the annotations that it would require.  
  > **username_2**: I agree that a neural net seems like overkill. That being said, this paper seems interesting: "Convolutional Neural Network Architecture for Geometric Matching" Ignacio Rocco, Relja Arandjelovic, Josef Sivic https://openaccess.thecvf.com/content_cvpr_2017/html/Rocco_Convolutional_Neural_Network_CVPR_2017_paper.html  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-03-01. Score: 5
  
  I think the second approach will be the best because it only requires that your training set is annotated with four labels for each of the four corners of the paper sheet.
  
  This is sort of the idea of a Region Proposal Network which is used in [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf) (section 3.1).
  
  [Here](https://github.com/pytorch/vision/blob/5e56575e688a85a3bc9dc3c97934dd864b65ce47/torchvision/models/detection/rpn.py#L88-L367) is a reference implementation of a Region Proposal Network in PyTorch from the [torchvision](https://github.com/pytorch/vision/) library. Notice how the network outputs `boxes` (in the `forward()` method) which is a tuple `(x1, y1, x2, y2)`. From these four coordinates, you could crop the image to the desired paper sheet region.
  
  
  ---
  
  ### Answer by username_4 on 2022-03-01. Score: 2
  
  You could try U-Net for approach 1.
  
  This is called the image-to-image translation problem in machine learning. You could see more architectures in this paper:
  <https://arxiv.org/pdf/2101.08629.pdf>
  
  > **Comments:**
  > **username_2**: A U-net wouldn't help a whole lot if it's being spatially transformed, since the skip connections wouldn't line up correctly between the input and output. That said, I guess a U-net is probably better than an autoencoder-like model without _any_ skip connections, so idk.  
  > **username_0**: @username_2 What kind of neural network would you use as a general tool for the general problem of image=in image=out, and you want the NN the learn transformations of images on thousands of examples? I would be interested in an answer about this, if you have one! Thanks  
  > **username_2**: @username_0 Accurate spatial transformations are something that current neural net architectures are just fundamentally unsuited to solve. U-nets connect pixels that are in the same location in the input and output, but with spatial transformations those pixels don't correspond to one another, so it's pointless. Other architectures can try to transform the image, but will lose a significant amount of detail (e.g. any text will become gibberish or disappear entirely). If you _really_ want to involve a neural net, use it to find the corners, and then use traditional methods to transform the image.  
  
  ---
  
  