Answers.Body: <p>If your PDF invoices have well-defined and clearly bordered tables, tools like <a href="https://github.com/atlanhq/camelot" rel="nofollow noreferrer">Camelot</a> and <a href="https://tabula.technology/" rel="nofollow noreferrer">Tabula</a> can directly handle table extraction without needing preliminary OCR. These tools can read the PDF’s internal structure to extract tables if the PDF is not scanned and already contains selectable text. Camelot provides two extraction modes—stream (for tables without borders) and lattice (for tables with borders). This flexibility makes it suitable for a variety of table structures.</p>
  <p>For scanned PDFs where text is represented as images, OCR (Optical Character Recognition) tools like <a href="https://en.wikipedia.org/wiki/Tesseract_(software)" rel="nofollow noreferrer">Tesseract</a> is essential for converting images of text into machine-readable text. Table extraction tools like Camelot and Tabula rely on having text data to work with. Tesseract can preprocess the PDF to convert scanned images into text, which can then be further processed by table extraction tools.</p>
  <p>Based on the final table extracted from an invoice PDF, you can break down the structured data into semantically meaningful chunks simply by rows and columns, and convert these chunks into vector representations such as Word2Vec, GloVe or more advanced models (BERT). Row chunks are flexible for querying and access, while column chunks can be used for analytical purposes or to perform operations like computing statistics. This is a type of semantic chunking following your reference's taxonomy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-12T01:11:37.153Z
Answers.LastActivityDate: 2024-08-12T01:11:37.153Z
Answers.UserDisplayName: cinch
Body: <p>I am new RAG Application. Currently I am working in a RAG application where I have to do Ingestion of PDF invoice document and have to fill-up some date from JSON structure and store in NOSQL DB.</p>
  <p>My PDF Invoice PDF is mostly contained tables based content. Either it is a single table or multiple table with rows and columns. Each data element I will fetch convert in JSON and store in NO SQL DB.</p>
  <p>Now I am confused for a PDF where mostly we have tables and box or place holder to content data in that case which chunking process I should use so that I can get most meaning full data which may be will store in vector database , and after that do a similarity search and use those data for standard JSON preparation.</p>
  <p>What I try :</p>
  <p>according to <a href="https://medium.com/@anuragmishra_27746/five-levels-of-chunking-strategies-in-rag-notes-from-gregs-video-7b735895694d" rel="nofollow noreferrer">https://medium.com/@anuragmishra_27746/five-levels-of-chunking-strategies-in-rag-notes-from-gregs-video-7b735895694d</a></p>
  <p>I was trying with Agentic Chunking, but the output was not satisfactory as in my case pdf is mostly contained table data and data in box , example as you can find in the attachment.</p>
  <p>Can any one suggest a suitable approach for chunking and embedding which may be I can use for this type of document where data are mostly given in format of table or box with some heading and then description as given in the attachment.
  Thanks in Advance....</p>
  <p>PDF Invoices are :
  <a href="https://i.sstatic.net/y6rP7C0w.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/y6rP7C0w.jpg" alt="enter image description here" /></a></p>
  
Tags: <ai-design><open-ai><embeddings><data-augmentation><embedded-design>
Title: RAG Pdf chunking and Vector Embeding
UserDisplayName: Sujoy
ThreadText: # RAG Pdf chunking and Vector Embeding
  
  (Asked by: username_0 on 2024-08-11)
  
  I am new RAG Application. Currently I am working in a RAG application where I have to do Ingestion of PDF invoice document and have to fill-up some date from JSON structure and store in NOSQL DB.
  
  My PDF Invoice PDF is mostly contained tables based content. Either it is a single table or multiple table with rows and columns. Each data element I will fetch convert in JSON and store in NO SQL DB.
  
  Now I am confused for a PDF where mostly we have tables and box or place holder to content data in that case which chunking process I should use so that I can get most meaning full data which may be will store in vector database , and after that do a similarity search and use those data for standard JSON preparation.
  
  What I try :
  
  according to <https://medium.com/@anuragmishra_27746/five-levels-of-chunking-strategies-in-rag-notes-from-gregs-video-7b735895694d>
  
  I was trying with Agentic Chunking, but the output was not satisfactory as in my case pdf is mostly contained table data and data in box , example as you can find in the attachment.
  
  Can any one suggest a suitable approach for chunking and embedding which may be I can use for this type of document where data are mostly given in format of table or box with some heading and then description as given in the attachment.
  Thanks in Advance....
  
  PDF Invoices are :
  [![enter image description here](https://i.sstatic.net/y6rP7C0w.jpg)](https://i.sstatic.net/y6rP7C0w.jpg)
  
  **Tags:** ai-design, open-ai, embeddings, data-augmentation, embedded-design
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-12. Score: 1
  
  If your PDF invoices have well-defined and clearly bordered tables, tools like [Camelot](https://github.com/atlanhq/camelot) and [Tabula](https://tabula.technology/) can directly handle table extraction without needing preliminary OCR. These tools can read the PDF’s internal structure to extract tables if the PDF is not scanned and already contains selectable text. Camelot provides two extraction modes—stream (for tables without borders) and lattice (for tables with borders). This flexibility makes it suitable for a variety of table structures.
  
  For scanned PDFs where text is represented as images, OCR (Optical Character Recognition) tools like [Tesseract](https://en.wikipedia.org/wiki/Tesseract_(software)) is essential for converting images of text into machine-readable text. Table extraction tools like Camelot and Tabula rely on having text data to work with. Tesseract can preprocess the PDF to convert scanned images into text, which can then be further processed by table extraction tools.
  
  Based on the final table extracted from an invoice PDF, you can break down the structured data into semantically meaningful chunks simply by rows and columns, and convert these chunks into vector representations such as Word2Vec, GloVe or more advanced models (BERT). Row chunks are flexible for querying and access, while column chunks can be used for analytical purposes or to perform operations like computing statistics. This is a type of semantic chunking following your reference's taxonomy.
  
  
  ---
  
  