Upload 5 files

Methodology and Privacy Protection.md

@@ -0,0 +1,65 @@
+# Synopsis Scorer: Methodology and Privacy Protection Strategy
+
+## Scoring Methodology
+
+The synopsis quality evaluation system employs a multi-dimensional approach to assess how effectively a synopsis captures and communicates the content of its source article:
+
+### Components (Total: 100 points)
+
+#### 1. **Content Coverage – 50 points**
+- **What it checks:** How well the synopsis captures the key ideas from the article.
+- **How it's done:**
+  - Uses the `all-MiniLM-L6-v2` SentenceTransformer model to convert both texts into vector form.
+  - Calculates the cosine similarity between the article and synopsis embeddings (range: 0 to 1).
+- **Scoring:** `similarity × 50` (e.g., 0.9 similarity = 45 points).
+- **Why it matters:** A strong synopsis reflects the main points from the original content.
+
+#### 2. **Clarity – 25 points**
+- **What it checks:** How clearly and precisely the synopsis is written.
+- **How it's done:**
+  - Calculates lexical diversity: `(unique words / total words) × 25`.
+- **Why it matters:** More diverse vocabulary indicates better language use and avoids repetition.
+
+#### 3. **Coherence – 25 points**
+- **What it checks:** How logically the synopsis is structured and whether it flows smoothly.
+- **How it's done:**
+  - Gives 5 points per sentence, up to a maximum of 5 sentences.
+- **Why it matters:** Clear, well-structured writing is easier to understand and follow.
+
+### Advanced Feedback
+
+In addition to the quantitative scoring, the system leverages the Gemma 3 4B LLM to provide qualitative feedback on synopsis quality. The model is guided through careful prompt engineering to focus on relevance, coverage, clarity, and coherence without storing or reproducing the original text content.
+
+## Privacy Protection Strategy
+
+The system implements a comprehensive data privacy approach to protect sensitive information:
+
+### Multi-Layer Anonymization
+
+1. **Named Entity Recognition**
+   - Uses spaCy's NER capabilities to identify and replace sensitive entities:
+     - PERSON: Individual names
+     - DATE: Temporal identifiers
+     - LOCATION/GPE: Geographic references
+     - ORG: Organization names
+
+2. **Regex Pattern Matching**
+   - Supplements NER with custom regular expressions to catch:
+     - Email addresses
+     - URLs and web links
+     - Phone numbers
+     - Identification numbers/codes
+
+3. **Privacy-Preserving LLM Integration**
+   - Applies anonymization before sending text to the LLM
+   - Uses a quantized model running locally to avoid data transmission to external APIs
+   - Implements character limits to prevent overloading and potential information leakage
+
+### System Design Considerations
+
+- **Local Processing**: All text processing occurs on the user's machine
+- **Access Control**: Token-based authentication restricts unauthorized access
+- **Data Minimization**: Preview displays only limited text portions
+- **Secure LLM Integration**: Carefully constructed prompts instruct the LLM to analyze without storing or reproducing sensitive content
+
+This privacy-first approach ensures that the system can provide valuable evaluation feedback while maintaining the confidentiality of sensitive information in both source articles and synopses.

README.md

@@ -1,19 +1,86 @@
----
-title: Synopsis Scorer
-emoji: 🚀
-colorFrom: red
-colorTo: red
-sdk: docker
-app_port: 8501
-tags:
-- streamlit
-pinned: false
-short_description: '🧠 Privacy-Preserving Synopsis Scorer Upload an article and '
----
-
-# Welcome to Streamlit!
-
-Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart:
-
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
+# Synopsis Scorer with Privacy Protection
+
+This application evaluates the quality of text synopses against their source content while maintaining privacy through robust text anonymization techniques.
+
+## Features
+
+- **Synopsis Quality Assessment**: Scores synopses based on content coverage, clarity, and coherence
+- **Privacy Protection**: Anonymizes sensitive information in both source articles and synopses
+- **LLM-Powered Feedback**: Provides qualitative feedback using Gemma 3 4B LLM
+- **User-Friendly Interface**: Built with Streamlit for easy interaction
+
+## Setup Instructions
+
+### Prerequisites
+
+- Python 3.8+
+- At least 4GB RAM (recommended for LLM inference)
+- 4GB disk space
+
+### Installation
+
+1. Clone this repository:
+   ```
+   git clone https://github.com/yourusername/synopsis-scorer.git
+   cd synopsis-scorer
+   ```
+
+2. Create a virtual environment:
+   ```
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. Install dependencies:
+   ```
+   pip install -r requirements.txt
+   ```
+
+4. Download the spaCy model:
+   ```
+   python -m spacy download en_core_web_sm
+   ```
+
+5. Download the Gemma model:
+   The application will automatically download the quantized Gemma model on first run, or you can manually download it:
+   ```
+   mkdir -p models
+
+   Download the model from this url:  https://huggingface.co/google/gemma-3-4b-it-qat-q4_0-gguf/resolve/main/gemma-3-4b-it-q4_0.gguf and place it in "models" folder.
+   ```
+
+### Running the Application
+
+1. Create a `.streamlit/secrets.toml` file with your access token:
+   ```
+   echo 'access_token = "your_secret_token"' > .streamlit/secrets.toml
+   echo 'hf_token = "your_huggingface_token"' > .streamlit/secrets.toml
+   ```
+
+2. Start the application:
+   ```
+   streamlit run app.py
+   ```
+
+3. Open your browser and go to `http://localhost:8501`
+
+## Usage
+
+1. Enter the access token to unlock the application
+2. Upload an article file (PDF or TXT format)
+3. Upload a synopsis file (TXT format)
+4. Click "Evaluate" to process and score the synopsis
+5. Review the scoring metrics and LLM feedback
+
+## Project Structure
+
+```
+synopsis-scorer/
+├── app.py                  # Main Streamlit application
+├── utils.py                # Utilities for text processing and scoring
+├── requirements.txt        # Python dependencies
+├── README.md               # This documentation
+├── .streamlit/
+│   └── secrets.toml        # Configuration secrets
+└── models/                 # Downloaded LLM models
+```

requirements.txt

@@ -1,3 +1,8 @@
-altair
-pandas
-streamlit
+streamlit==1.31.0
+PyMuPDF==1.23.8
+numpy==1.26.2
+scikit-learn==1.3.2
+sentence-transformers==2.2.2
+spacy==3.7.2
+llama-cpp-python==0.2.38
+en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

streamlit_app.py

@@ -0,0 +1,76 @@
+import streamlit as st
+from utils import extract_text, anonymize_text, score_synopsis
+from llama_cpp import Llama
+import os
+
+st.set_page_config(page_title="Synopsis Scorer", layout="wide")
+
+# --- Access Control ---
+TOKEN = st.secrets.get("access_token")
+user_token = st.text_input("Enter Access Token to Continue", type="password")
+if user_token != TOKEN:
+    st.warning("Please enter a valid access token.")
+    st.stop()
+
+# --- Hugging Face Token Configuration ---
+hf_token = st.secrets.get("hf_token") if "hf_token" in st.secrets else os.environ.get("HF_TOKEN")
+if not hf_token and not os.path.exists("models/gemma-3-4b-it-q4_0.gguf"):
+    st.warning("Hugging Face token not found. Please add it to your secrets or environment variables.")
+    hf_token = st.text_input("Enter your Hugging Face token:", type="password")
+
+
+# --- File Upload ---
+st.title("📘 Synopsis Scorer with Privacy Protection")
+article_file = st.file_uploader("Upload the Article (.pdf/.txt)", type=["pdf", "txt"])
+synopsis_file = st.file_uploader("Upload the Synopsis (.txt)", type=["txt"])
+
+if article_file and synopsis_file:
+    with st.spinner("Reading files..."):
+        article = extract_text(article_file)
+        synopsis = extract_text(synopsis_file)
+
+    st.subheader("Preview")
+    st.text_area("Article", article[:1000] + "...", height=200)
+    st.text_area("Synopsis", synopsis, height=150)
+
+    if st.button("Evaluate"):
+        with st.spinner("Scoring..."):
+            scores = score_synopsis(article, synopsis)
+
+            # Anonymization
+            article_anon = anonymize_text(article)
+            synopsis_anon = anonymize_text(synopsis)
+            
+            article_limit = 350000 # max_chars = 128000 * 3.5 (approx_chars_per_token) ≈ 448,000 characters;  448,000 - 98000(space for synopsis) = 350000
+            
+            # LLM feedback
+            try:
+                llm = Llama.from_pretrained(
+                    repo_id="google/gemma-3-4b-it-qat-q4_0-gguf",
+                    filename="gemma-3-4b-it-q4_0.gguf"
+                )
+                prompt = (
+                    "You are an expert writing evaluator. The user has uploaded two text documents: "
+                    "1) a short synopsis, and 2) a longer article (source content). "
+                    "Without copying or storing the full content, analyze the synopsis and evaluate its quality in comparison to the article. "
+                    "Assess it on the basis of relevance, coverage, clarity, and coherence.\n\n"
+                    "Return:\n- A score out of 100\n- 2 to 3 lines of qualitative feedback\n\n"
+                    f"Here is the source article:\n{article_anon[:article_limit]}\n\nHere is the synopsis:\n{synopsis_anon}"
+                )
+                result = llm.create_chat_completion(messages=[{"role": "user", "content": [{"type": "text", "text": prompt}]}])
+                feedback = result["choices"][0]["message"]["content"]
+            except Exception as e:
+                feedback = "LLM feedback not available: " + str(e)
+
+        st.success("Evaluation Complete ✅")
+
+        st.metric("Total Score", f"{scores['total']} / 100")
+        st.progress(scores["total"] / 100)
+
+        st.subheader("Score Breakdown")
+        st.write(f"📘 Content Coverage: {scores['content_coverage']} / 50")
+        st.write(f"🧠 Clarity: {scores['clarity']} / 25")
+        st.write(f"🔗 Coherence: {scores['coherence']} / 25")
+
+        st.subheader("LLM Feedback")
+        st.write(feedback)

utils.py

@@ -0,0 +1,65 @@
+import re 
+import fitz
+import numpy as np
+from sklearn.metrics.pairwise import cosine_similarity
+from sentence_transformers import SentenceTransformer
+import spacy 
+
+
+# Load the English NLP model and the SentenceTransformer model
+nlp = spacy.load("en_core_web_sm")
+model = SentenceTransformer('all-MiniLM-L6-v2')
+
+def extract_text(file):
+    if file.name.endswith(".pdf"):
+        doc = fitz.open(stream=file.read(), filetype="pdf")
+        return "\n".join([page.get_text() for page in doc])
+    else:
+        return file.read().decode("utf-8")
+    
+def anonymize_text(text):
+    doc = nlp(text)
+    #Collect spaCy-detected entities
+    replacements = []
+    for ent in doc.ents:
+        if ent.label_ == "PERSON":
+            replacements.append((ent.start_char, ent.end_char, "PERSON"))
+        elif ent.label_ == "DATE":
+            replacements.append((ent.start_char, ent.end_char, "DATE"))
+        elif ent.label_ in ["GPE", "LOC"]:
+            replacements.append((ent.start_char, ent.end_char, "LOCATION"))
+        elif ent.label_ == "ORG":
+            replacements.append((ent.start_char, ent.end_char, "ORG"))
+    
+    #Add regex-based matches for things spaCy misses
+    regex_patterns = [
+        (r"\b[\w\.-]+@[\w\.-]+\.\w+\b", "EMAIL"),     # Email
+        (r"https?://\S+|www\.\S+", "URL"),            # URLs
+        (r"\b\d{10}\b", "PHONE"),                     # 10-digit phone numbers
+        (r"\b[A-Z]{2,}\d{6,}\b", "ID"),               # Generic IDs (e.g., AA123456)
+    ]
+    for pattern, label in regex_patterns:
+        for match in re.finditer(pattern, text):
+            replacements.append((match.start(), match.end(), label))
+    
+    replacements.sort(reverse=True)
+    for start, end, replacement in replacements:
+        text = text[:start] + f"[{replacement}]" + text[end:]  # Adding brackets for clarity
+    
+    return text
+
+def score_synopsis(article, synopsis):
+    embeddings = model.encode([article, synopsis])
+    similarity = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
+
+    content_coverage = similarity * 50
+    clarity = (len(set(synopsis.split())) / max(len(synopsis.split()), 1)) * 25
+    coherence = min(25, 5 * (len(synopsis.split(".")) - 1))
+
+    total = content_coverage + clarity + coherence
+    return {
+        "total": round(total, 2),
+        "content_coverage": round(content_coverage, 2),
+        "clarity": round(clarity, 2),
+        "coherence": round(coherence, 2)
+    }