Answers.Body: <p>Yes, it is possible, CNNs can also be used for OCR (see the <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST task</a> and <a href="https://nanonets.com/blog/deep-learning-ocr/" rel="nofollow noreferrer">this blog</a>), although it's not the common way for OCR because it is considered a bit overkill and inefficient. Furthermore, OCR is considered a solved problem already and don't need deep learning to do well, unless perhaps under unfavorable conditions like dark lighting, complex background of the document, weird fonts, occlusion of text, etc.</p>
  <p>Breaking down your questions:</p>
  <ol>
  <li>a CNN can likely classify an image as &quot;this image has a word document in it&quot; vs &quot;this image does not have a word document in it&quot;. The word document itself, and the outline of the words, should form distinct enough features for the CNN to discriminate against</li>
  <li>a CNN can likely identify the vendor based on the logo of the vendor if you have enough training images of it at various distortions (angles, occlusions, different lighting, etc)</li>
  <li>a CNN can likely do OCR well enough to extract the text, and then an entity extraction model can identify the vendor from the words. However, this approach would be uncommon as OCR is already considered a solved problem. But a neural network can potentially help preprocessing difficult OCR images (rotating image, identifying text boundaries, fix lighting, perform super-resolution, etc) to increase OCR accuracy</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T17:26:03.690Z
Answers.Text: What I'd like to do is identify the vendor purely based on the image, not using any OCR. So I'd have folders for vendor1, vendor2, vendor3, etc. with the first page of dozens of invoices for each to train the CNN. Just not sure if an image of a document has enough nuances for the CNN to distinguish, especially since these are not different document types, they're all invoices, just different vendors.
Answers.UserDisplayName: Rocket04
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-18T22:43:29.103Z
Answers.Text: It's theoretically possible for CNN to do well, but I think in practice with commodity hardware and the CNN architecture we have today, it will perform poorly. Would be better to use OCR, then use fuzzy keyword matching to see if the receipt mentions a vendor names. You'd only need to build a database of vendor names with this method, which is much easier than designing + training a neural network model.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T20:42:41.010Z
Answers.LastActivityDate: 2020-12-14T20:42:41.010Z
Answers.UserDisplayName: user3667125
Body: <p>I know convolutional neural networks are commonly used for image recognition, but I was wondering if they would be able to distinguish between predominantly text-based documents vs something like objects. For example, if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image? If not, is there a different AI technique better suited that is purely image-based, or would it require OCR and leveraging the text in the invoice?</p>
  <p>Update: based on a comment, my ask my not be clear. I'm not trying to see if the CNN can differentiate between a document (mostly text based image) and a photo image. I want to know if based on a gif/jpeg/png of a document (no OCR performed) a CNN would be able to classify the documents, which basically could be used as a means of identifying the vendor.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-13T12:20:45.523Z
Comments.Text: When you say "be able to distinguish between predominantly text-based documents vs something like objects", are you asking if CNNs would be able to distinguish between 1. images that contain predominantly text and 2. images that contain predominantly objects? Then you say "if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image?", here "first page of invoices matched to a vendor name", I assume that invoices are examples of documents that contain mainly text...
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-13T12:21:37.553Z
Comments.Text: but then you say "to predict the vendor based on an image", here "image" refers to the invoice, right? So, are you asking whether CNNs can actually perform classification of text-based documents?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T14:17:00.320Z
Comments.Text: @nbro Yes, I'm asking if they can perform classification of text-based documents, but without relying on OCR. So basically, classify based on how the document "looks". So, there's the same logo top left, similar position of text, etc. Granted, not two invoices will be the same, date will change, etc. Just wondering if a predominantly text based image provides enough nuance for a CNN to classify or not.
Comments.UserDisplayName: Rocket04
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T14:17:09.273Z
Comments.Text: I will clarify the question.
Comments.UserDisplayName: Rocket04
Tags: <convolutional-neural-networks><image-recognition><optical-character-recognition>
Title: Can a convolutional neural network classify text document images?
UserDisplayName: Rocket04
ThreadText: # Can a convolutional neural network classify text document images?
  
  (Asked by: username_0 on 2020-12-13)
  
  I know convolutional neural networks are commonly used for image recognition, but I was wondering if they would be able to distinguish between predominantly text-based documents vs something like objects. For example, if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image? If not, is there a different AI technique better suited that is purely image-based, or would it require OCR and leveraging the text in the invoice?
  
  Update: based on a comment, my ask my not be clear. I'm not trying to see if the CNN can differentiate between a document (mostly text based image) and a photo image. I want to know if based on a gif/jpeg/png of a document (no OCR performed) a CNN would be able to classify the documents, which basically could be used as a means of identifying the vendor.
  
  **Tags:** convolutional-neural-networks, image-recognition, optical-character-recognition
  > **Comments:**
  > **username_1**: When you say "be able to distinguish between predominantly text-based documents vs something like objects", are you asking if CNNs would be able to distinguish between 1. images that contain predominantly text and 2. images that contain predominantly objects? Then you say "if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image?", here "first page of invoices matched to a vendor name", I assume that invoices are examples of documents that contain mainly text...  
  > **username_1**: but then you say "to predict the vendor based on an image", here "image" refers to the invoice, right? So, are you asking whether CNNs can actually perform classification of text-based documents?  
  > **username_0**: @nbro Yes, I'm asking if they can perform classification of text-based documents, but without relying on OCR. So basically, classify based on how the document "looks". So, there's the same logo top left, similar position of text, etc. Granted, not two invoices will be the same, date will change, etc. Just wondering if a predominantly text based image provides enough nuance for a CNN to classify or not.  
  > **username_0**: I will clarify the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-14. Score: 0 | Accepted answer
  
  Yes, it is possible, CNNs can also be used for OCR (see the [MNIST task](http://yann.lecun.com/exdb/mnist/) and [this blog](https://nanonets.com/blog/deep-learning-ocr/)), although it's not the common way for OCR because it is considered a bit overkill and inefficient. Furthermore, OCR is considered a solved problem already and don't need deep learning to do well, unless perhaps under unfavorable conditions like dark lighting, complex background of the document, weird fonts, occlusion of text, etc.
  
  Breaking down your questions:
  
  1. a CNN can likely classify an image as "this image has a word document in it" vs "this image does not have a word document in it". The word document itself, and the outline of the words, should form distinct enough features for the CNN to discriminate against
  2. a CNN can likely identify the vendor based on the logo of the vendor if you have enough training images of it at various distortions (angles, occlusions, different lighting, etc)
  3. a CNN can likely do OCR well enough to extract the text, and then an entity extraction model can identify the vendor from the words. However, this approach would be uncommon as OCR is already considered a solved problem. But a neural network can potentially help preprocessing difficult OCR images (rotating image, identifying text boundaries, fix lighting, perform super-resolution, etc) to increase OCR accuracy
  
  > **Comments:**
  > **username_0**: What I'd like to do is identify the vendor purely based on the image, not using any OCR. So I'd have folders for vendor1, vendor2, vendor3, etc. with the first page of dozens of invoices for each to train the CNN. Just not sure if an image of a document has enough nuances for the CNN to distinguish, especially since these are not different document types, they're all invoices, just different vendors.  
  > **username_2**: It's theoretically possible for CNN to do well, but I think in practice with commodity hardware and the CNN architecture we have today, it will perform poorly. Would be better to use OCR, then use fuzzy keyword matching to see if the receipt mentions a vendor names. You'd only need to build a database of vendor names with this method, which is much easier than designing + training a neural network model.  
  
  ---
  
  