Upload 5 files

DualTextOCRFusion

.gitignore

@@ -0,0 +1,55 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environment
+venv/
+env/
+.venv/
+.env/
+ENV/
+.env.bak/
+*.env
+__pycache__
+
+# VS Code
+.vscode/
+.history/
+
+# PyCharm
+.idea/
+
+# Jupyter Notebook
+.ipynb_checkpoints/
+
+# Logs
+*.log
+
+# Mac OS files
+.DS_Store
+
+# Streamlit Cache (Optional)
+streamlit_cache/

README.md

@@ -1,13 +1,150 @@
----
-title: DualTextOCRFusion
-emoji: ⚡
-colorFrom: gray
-colorTo: purple
-sdk: streamlit
-sdk_version: 1.38.0
-app_file: app.py
-pinned: false
-license: mit
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🔍 DualTextOCRFusion
+
+**DualTextOCRFusion** is a web-based Optical Character Recognition (OCR) application that allows users to upload images containing both Hindi and English text, extract the text, and search for keywords within the extracted text. The app uses advanced models like **ColPali’s Byaldi + Qwen2-VL** or **General OCR Theory (GOT)** for multilingual text extraction.
+
+## Features
+
+- **Multilingual OCR**: Extract text from images containing both **Hindi** and **English**.
+- **Keyword Search**: Search for specific keywords in the extracted text.
+- **User-Friendly Interface**: Simple, intuitive interface for easy image uploading and searching.
+- **Deployed Online**: Accessible through a live URL for easy use.
+
+## Technologies Used
+
+- **Python**: Backend logic.
+- **Streamlit**: For building the web interface.
+- **Huggingface Transformers**: For integrating OCR models (Qwen2-VL or GOT).
+- **PyTorch**: For deep learning inference.
+- **Pytesseract**: Optional OCR engine.
+- **OpenCV**: For image preprocessing.
+
+## Project Structure
+
+```
+DualTextOCRFusion/
+│
+├── app.py                 # Main Streamlit application
+├── ocr.py                 # Handles OCR extraction using the selected model
+├── .gitignore             # Files and directories to ignore in Git
+├── .streamlit/
+│   └── config.toml        # Streamlit theme configuration
+├── requirements.txt       # Dependencies for the project
+└── README.md              # This file
+```
+
+## How to Run Locally
+
+### Prerequisites
+
+- Python 3.8 or above installed on your machine.
+- Tesseract installed for using `pytesseract` (optional if using Huggingface models). You can download Tesseract from [here](https://github.com/tesseract-ocr/tesseract).
+
+### Steps
+
+1. **Clone the Repository**:
+
+   ```bash
+   git clone https://github.com/yourusername/dual-text-ocr-fusion.git
+   cd dual-text-ocr-fusion
+   ```
+
+2. **Install Dependencies**:
+
+   Make sure you have the required dependencies by running the following:
+
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+3. **Run the Application**:
+
+   Start the Streamlit app by running the following command:
+
+   ```bash
+   streamlit run app.py
+   ```
+
+4. **Open the App**:
+
+   Once the server starts, the app will be available in your browser at:
+
+   ```
+   http://localhost:8501
+   ```
+
+### Usage
+
+1. **Upload an Image**: Upload an image containing Hindi and English text in formats like JPG, JPEG, or PNG.
+2. **View Extracted Text**: The app will extract and display the text from the image.
+3. **Search for Keywords**: Enter any keyword to search within the extracted text.
+
+## Deployment
+
+The app is deployed on **Streamlit Sharing** and can be accessed via the live URL:
+
+**[Live Application](https://your-app-link.streamlit.app)**
+
+## Customization
+
+### Changing the OCR Model
+
+By default, the app uses the **Qwen2-VL** model, but you can switch to the **General OCR Theory (GOT)** model by editing the `ocr.py` file.
+
+- **For Qwen2-VL**:
+  
+  ```python
+  from ocr import extract_text_byaldi
+  ```
+
+- **For General OCR Theory (GOT)**:
+  
+  ```python
+  from ocr import extract_text_got
+  ```
+
+### Custom UI Theme
+
+You can customize the look and feel of the application by modifying the `.streamlit/config.toml` file. Adjust colors, fonts, and layout options to suit your preferences.
+
+## Example Images
+
+Here are some sample images you can use to test the OCR functionality:
+
+1. **Sample 1**: A document with mixed Hindi and English text.
+2. **Sample 2**: An image with only Hindi text for multilingual OCR testing.
+
+## Contributing
+
+If you'd like to contribute to this project, feel free to fork the repository and submit a pull request. Follow these steps:
+
+1. Fork the project.
+2. Create a feature branch:
+
+   ```bash
+   git checkout -b feature-branch
+   ```
+
+3. Commit your changes:
+
+   ```bash
+   git commit -am 'Add new feature'
+   ```
+
+4. Push to the branch:
+
+   ```bash
+   git push origin feature-branch
+   ```
+
+5. Open a pull request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Credits
+
+- **Streamlit**: For the easy-to-use web interface.
+- **Huggingface Transformers**: For the powerful OCR models.
+- **Tesseract**: For optional OCR functionality.
+- **ColPali & GOT Models**: For the multilingual OCR support.

app.py

@@ -0,0 +1,72 @@
+import streamlit as st
+from ocr_cpu import extract_text_got  # The updated OCR function
+import json
+
+# --- UI Styling ---
+st.set_page_config(page_title="DualTextOCRFusion",
+                   layout="centered", page_icon="🔍")
+
+st.markdown(
+    """
+    <style>
+    .reportview-container {
+        background: #f4f4f4;
+    }
+    .sidebar .sidebar-content {
+        background: #e0e0e0;
+    }
+    h1 {
+        color: #007BFF;
+    }
+    .upload-btn {
+        background-color: #007BFF;
+        color: white;
+        padding: 10px;
+        border-radius: 5px;
+        text-align: center;
+    }
+    </style>
+    """, unsafe_allow_html=True
+)
+
+# --- Title ---
+st.title("🔍 DualTextOCRFusion")
+st.write("Upload an image with **Hindi** and **English** text to extract and search for keywords.")
+
+# --- Image Upload Section ---
+uploaded_file = st.file_uploader(
+    "Choose an image file", type=["jpg", "jpeg", "png"])
+
+if uploaded_file is not None:
+    st.image(uploaded_file, caption='Uploaded Image', use_column_width=True)
+
+    # Extract text from the image using the selected OCR function (GOT)
+    with st.spinner("Extracting text using the model..."):
+        try:
+            extracted_text = extract_text_got(
+                uploaded_file)  # Pass uploaded_file directly
+            if not extracted_text.strip():
+                st.warning("No text extracted from the image.")
+        except Exception as e:
+            st.error(f"Error during text extraction: {str(e)}")
+            extracted_text = ""
+
+    # Display extracted text
+    st.subheader("Extracted Text")
+    st.text_area("Text", extracted_text, height=250)
+
+    # Save extracted text for search
+    if extracted_text:
+        with open("extracted_text.json", "w") as json_file:
+            json.dump({"text": extracted_text}, json_file)
+
+        # --- Keyword Search ---
+        st.subheader("Search for Keywords")
+        keyword = st.text_input(
+            "Enter a keyword to search in the extracted text")
+
+        if keyword:
+            if keyword.lower() in extracted_text.lower():
+                st.success(f"Keyword **'{keyword}'** found in the text!")
+            else:
+                st.error(f"Keyword **'{keyword}'** not found.")

ocr_cpu.py

@@ -0,0 +1,97 @@
+import os
+from transformers import AutoModel, AutoTokenizer
+import torch
+
+# Load model and tokenizer
+model_name = "ucaslcl/GOT-OCR2_0"
+tokenizer = AutoTokenizer.from_pretrained(
+    model_name, trust_remote_code=True, return_tensors='pt'
+)
+
+# Load the model
+model = AutoModel.from_pretrained(
+    model_name,
+    trust_remote_code=True,
+    low_cpu_mem_usage=True,
+    use_safetensors=True,
+    pad_token_id=tokenizer.eos_token_id,
+)
+
+# Ensure the model is in evaluation mode and loaded on CPU
+device = torch.device("cpu")
+dtype = torch.float32  # Use float32 on CPU
+model = model.eval().to(device)
+
+# OCR function
+
+
+def extract_text_got(uploaded_file):
+    """Use GOT-OCR2.0 model to extract text from the uploaded image."""
+    try:
+        temp_file_path = 'temp_image.jpg'
+        with open(temp_file_path, 'wb') as temp_file:
+            temp_file.write(uploaded_file.read())  # Save file
+
+        # OCR attempts
+        ocr_types = ['ocr', 'format']
+        fine_grained_options = ['ocr', 'format']
+        color_options = ['red', 'green', 'blue']
+        box = [10, 10, 100, 100]  # Example box for demonstration
+        multi_crop_types = ['ocr', 'format']
+
+        results = []
+
+        # Run the model without autocast (not necessary for CPU)
+        for ocr_type in ocr_types:
+            with torch.no_grad():
+                outputs = model.chat(
+                    tokenizer, temp_file_path, ocr_type=ocr_type
+                )
+                if isinstance(outputs, list) and outputs[0].strip():
+                    return outputs[0].strip()  # Return if successful
+                results.append(outputs[0].strip() if outputs else "No result")
+
+        # Try FINE-GRAINED OCR with box options
+        for ocr_type in fine_grained_options:
+            with torch.no_grad():
+                outputs = model.chat(
+                    tokenizer, temp_file_path, ocr_type=ocr_type, ocr_box=box
+                )
+                if isinstance(outputs, list) and outputs[0].strip():
+                    return outputs[0].strip()  # Return if successful
+                results.append(outputs[0].strip() if outputs else "No result")
+
+        # Try FINE-GRAINED OCR with color options
+        for ocr_type in fine_grained_options:
+            for color in color_options:
+                with torch.no_grad():
+                    outputs = model.chat(
+                        tokenizer, temp_file_path, ocr_type=ocr_type, ocr_color=color
+                    )
+                    if isinstance(outputs, list) and outputs[0].strip():
+                        return outputs[0].strip()  # Return if successful
+                    results.append(outputs[0].strip()
+                                   if outputs else "No result")
+
+        # Try MULTI-CROP OCR
+        for ocr_type in multi_crop_types:
+            with torch.no_grad():
+                outputs = model.chat_crop(
+                    tokenizer, temp_file_path, ocr_type=ocr_type
+                )
+                if isinstance(outputs, list) and outputs[0].strip():
+                    return outputs[0].strip()  # Return if successful
+                results.append(outputs[0].strip() if outputs else "No result")
+
+        # If no text was extracted
+        if all(not text for text in results):
+            return "No text extracted."
+        else:
+            return "\n".join(results)
+
+    except Exception as e:
+        return f"Error during text extraction: {str(e)}"
+
+    finally:
+        if os.path.exists(temp_file_path):
+            os.remove(temp_file_path)

requirements.txt

@@ -0,0 +1,14 @@
+torch==2.0.1
+torchvision==0.15.2
+transformers==4.37.2
+megfile==3.1.2
+tiktoken
+verovio
+opencv-python
+cairosvg
+accelerate
+numpy==1.26.4
+loadimg
+pillow
+markdown
+shutils