Spaces:

ghostai1
/

internalRAGCX

Sleeping

App Files Files Community

ghostai1 commited on 8 days ago

Commit

aaeb925

verified ·

1 Parent(s): 209e38b

Update README.md

Browse files

Files changed (1) hide show

README.md +74 -307

README.md CHANGED Viewed

@@ -6,343 +6,110 @@ colorFrom: purple
 colorTo: green
 short_description: CX AI LLM
 ---
-title: Customer Experience Bot Demo emoji: 🤖 colorFrom: blue colorTo: purple sdk: gradio sdk_version: "4.44.0" app_file: app.py pinned: false
-Customer Experience Bot Demo
-A cutting-edge Retrieval-Augmented Generation (RAG) and Context-Augmented Generation (CAG) powered Customer Experience (CX) bot, deployed on Hugging Face Spaces (free tier). Architected with over 5 years of AI expertise since 2020, this demo leverages advanced Natural Language Processing (NLP) pipelines to deliver high-fidelity, multilingual CX solutions for enterprise-grade applications in SaaS, HealthTech, FinTech, and eCommerce. The system showcases robust data preprocessing for call center datasets, integrating state-of-the-art technologies like Pandas for data wrangling, Hugging Face Transformers for embeddings, FAISS for vectorized retrieval, and FastAPI-compatible API design principles for scalable inference.
-Technical Architecture
-Retrieval-Augmented Generation (RAG) Pipeline
-The core of this CX bot is a RAG framework, designed to fuse retrieval and generation for contextually relevant responses. The pipeline employs:
-Hugging Face Transformers: Utilizes all-MiniLM-L6-v2, a lightweight Sentence-BERT model (~80MB), fine-tuned for semantic embeddings, to encode call center FAQs into dense vectors. This ensures efficient, high-dimensional representation of query semantics.
-FAISS (CPU): Implements a FAISS IndexFlatL2 for similarity search, enabling rapid retrieval of top-k FAQs (default k=2) via L2 distance metrics. FAISS’s CPU optimization ensures free-tier compatibility while maintaining sub-millisecond retrieval latency.
-Rule-Based Generation: Bypasses heavy LLMs (e.g., GPT-2) for free-tier constraints, using retrieved FAQ answers directly, achieving a simulated 95% accuracy while minimizing compute overhead.
-Context-Augmented Generation (CAG) Integration
-Building on RAG, the system incorporates CAG principles by enriching retrieved contexts with metadata (e.g., call_id, language) from call center CSVs. This contextual augmentation enhances response relevance, particularly for multilingual CX (e.g., English, Spanish), ensuring the bot adapts to diverse enterprise needs.
-Call Center Data Preprocessing with Pandas
-The bot ingests raw call center CSVs, which are often riddled with junk data (nulls, duplicates, malformed entries). Leveraging Pandas, the preprocessing pipeline:
-Data Ingestion: Parses CSVs with pd.read_csv, using io.StringIO for embedded data, with explicit quotechar and escapechar to handle complex strings.
-Junk Data Cleanup:
-Null Handling: Drops rows with missing question or answer using df.dropna().
-Duplicate Removal: Eliminates redundant FAQs via df[~df['question'].duplicated()].
-Short Entry Filtering: Excludes questions <10 chars or answers <20 chars with df[(df['question'].str.len() >= 10) & (df['answer'].str.len() >= 20)].
-Malformed Detection: Uses regex ([!?]{2,}|\b(Invalid|N/A)\b) to filter invalid questions.
-Standardization: Normalizes text (e.g., mo to month) and fills missing language with en.
-Output: Generates cleaned_call_center_faqs.csv for downstream modeling, with detailed cleanup stats (e.g., nulls, duplicates removed).
-Enterprise-Grade Modeling Compatibility
-The cleaned CSV is optimized for:
-Amazon SageMaker: Ready for training BERT-based models (e.g., bert-base-uncased) for intent classification or FAQ retrieval, deployable via SageMaker JumpStart.
-Azure AI: Compatible with Azure Machine Learning pipelines for fine-tuning models like DistilBERT in Azure Blob Storage, enabling scalable CX automation.
-LLM Integration: While not used in this free-tier demo, the cleaned data supports fine-tuning LLMs (e.g., distilgpt2) for generative tasks, leveraging your FastAPI experience for API-driven inference.
-Performance Monitoring and Visualization
-The bot includes a performance monitoring suite:
-Latency Tracking: Measures embedding, retrieval, and generation times using time.perf_counter(), reported in milliseconds.
-Accuracy Metrics: Simulates retrieval accuracy (95% if FAQs retrieved, 0% otherwise) for demo purposes.
-Visualization: Uses Matplotlib and Seaborn to plot a dual-axis chart (rag_plot.png):
-Bar Chart: Latency (ms) per stage (Embedding, Retrieval, Generation).
-Line Chart: Accuracy (%) per stage, with a muted palette for professional aesthetics.
-Gradio Interface for Interactive CX
-The bot is deployed via Gradio, providing a user-friendly interface:
-Input: Text query field for user inputs (e.g., “How do I reset my password?”).
-Outputs:
-Bot response (e.g., “Go to the login page, click ‘Forgot Password,’...”).
-Retrieved FAQs with question-answer pairs.
-Cleanup stats (e.g., “Cleaned FAQs: 6; removed 4 junk entries”).
-RAG pipeline plot for latency and accuracy.
-Styling: Custom dark theme CSS (#2a2a2a background, blue buttons) for a sleek, enterprise-ready UI.
-Setup
-Clone this repository to a Hugging Face Space (free tier, public).
-Add requirements.txt with dependencies (gradio==4.44.0, pandas==2.2.3, etc.).
-Upload app.py (embeds call center FAQs for seamless deployment).
-Configure to run with Python 3.9+, CPU hardware (no GPU).
-Usage
-Query: Enter a question in the Gradio UI (e.g., “How do I reset my password?”).
-Output:
-Response: Contextually relevant answer from retrieved FAQs.
-Retrieved FAQs: Top-k question-answer pairs.
-Cleanup Stats: Detailed breakdown of junk data removal (nulls, duplicates, short entries, malformed).
-RAG Plot: Visual metrics for latency and accuracy.
-Example:
-Query: “How do I reset my password?”
-Response: “Go to the login page, click ‘Forgot Password,’ and follow the email instructions.”
-Cleanup Stats: “Cleaned FAQs: 6; removed 4 junk entries: 2 nulls, 1 duplicates, 1 short, 0 malformed”
-Call Center Data Cleanup
-Preprocessing Pipeline:
-Null Handling: Eliminates incomplete entries with df.dropna().
-Duplicate Removal: Ensures uniqueness via df[~df['question'].duplicated()].
-Short Entry Filtering: Maintains quality with length-based filtering.
-Malformed Detection: Uses regex to identify and remove invalid queries.
-Standardization: Normalizes text and metadata for consistency.
-Impact: Produces high-fidelity FAQs for RAG/CAG pipelines, critical for call center CX automation.
-Modeling Output: The cleaned cleaned_call_center_faqs.csv is ready for:
-SageMaker: Fine-tuning BERT models for intent classification or FAQ retrieval.
-Azure AI: Training DistilBERT in Azure ML for scalable CX automation.
-LLM Fine-Tuning: Supports advanced generative tasks with LLMs via FastAPI endpoints.
-Technical Details
-Stack:
-Pandas: Data wrangling and preprocessing for call center CSVs.
-Hugging Face Transformers: all-MiniLM-L6-v2 for semantic embeddings.
-FAISS: Vectorized similarity search with L2 distance metrics.
-Gradio: Interactive UI for real-time CX demos.
-Matplotlib/Seaborn: Performance visualization with dual-axis plots.
-FastAPI Compatibility: Designed with API-driven inference in mind, leveraging your experience with FastAPI for scalable deployments (e.g., RESTful endpoints for RAG inference).
-Free Tier Optimization: Lightweight with CPU-only dependencies, no GPU required.
-Extensibility: Ready for integration with enterprise CRMs (e.g., Salesforce) via FastAPI, and cloud deployments on AWS Lambda or Azure Functions.
-Purpose
-This demo showcases expertise in AI-driven CX automation, with a focus on call center data quality, built on over 5 years of experience in AI, NLP, and enterprise-grade deployments. It demonstrates the power of RAG and CAG pipelines, Pandas-based data preprocessing, and scalable modeling for SageMaker and Azure AI, making it ideal for advanced CX solutions in call center environments.
-Future Enhancements
-LLM Integration: Incorporate distilgpt2 or t5-small (from your past projects) for generative responses, fine-tuned on cleaned call center data.
-FastAPI Deployment: Expose RAG pipeline via FastAPI endpoints for production-grade inference.
-Multilingual Scaling: Expand language support (e.g., French, German) using Hugging Face’s multilingual models.
-Real-Time Monitoring: Add Prometheus metrics for latency/accuracy in production environments.

 colorTo: green
 short_description: CX AI LLM
 ---
+# Internal RAG CX Data Preprocessing Demo
+A robust data preprocessing pipeline for Retrieval-Augmented Generation (RAG) and Context-Augmented Generation (CAG) systems, deployed on Hugging Face as a Model repository (free tier). Built with over 5 years of AI expertise since 2020, this demo focuses on cleaning and preparing call center datasets for enterprise-grade CX applications in SaaS, HealthTech, FinTech, and eCommerce. It integrates advanced data wrangling with Pandas, ensuring high-quality FAQs for downstream RAG/CAG pipelines, and is compatible with Amazon SageMaker and Azure AI for scalable modeling.
+## Technical Architecture
+### Data Preprocessing Pipeline
+The core of this demo is a comprehensive data preprocessing pipeline designed to clean raw call center datasets:
+- **Data Ingestion**:
+  - Parses CSVs with `pd.read_csv`, using `io.StringIO` for embedded data, with explicit `quotechar` and `escapechar` to handle complex strings.
+  - Handles datasets with columns: `call_id`, `question`, `answer`, `language`.
+- **Junk Data Cleanup**:
+  - **Null Handling**: Drops rows with missing `question` or `answer` using `df.dropna()`.
+  - **Duplicate Removal**: Eliminates redundant FAQs via `df[~df['question'].duplicated()]`.
+  - **Short Entry Filtering**: Excludes questions <10 chars or answers <20 chars with `df[(df['question'].str.len() >= 10) & (df['answer'].str.len() >= 20)]`.
+  - **Malformed Detection**: Uses regex (`[!?]{2,}|\b(Invalid|N/A)\b`) to filter invalid questions.
+  - **Standardization**: Normalizes text (e.g., "mo" to "month") and fills missing `language` with "en".
+- **Output**:
+  - Generates `cleaned_call_center_faqs.csv` for downstream modeling.
+  - Provides cleanup stats: nulls removed, duplicates removed, short entries filtered, malformed entries detected.
+### Enterprise-Grade Modeling Compatibility
+The cleaned dataset is optimized for:
+- **Amazon SageMaker**: Ready for training BERT-based models (e.g., `bert-base-uncased`) for intent classification or FAQ retrieval, deployable via SageMaker JumpStart.
+- **Azure AI**: Compatible with Azure Machine Learning pipelines for fine-tuning models like DistilBERT in Azure Blob Storage, enabling scalable CX automation.
+- **LLM Integration**: Supports fine-tuning LLMs (e.g., `distilgpt2`) for generative tasks, leveraging your FastAPI experience for API-driven inference.
+## Performance Monitoring and Visualization
+The demo includes a performance monitoring suite:
+- **Processing Time Tracking**: Measures data ingestion, cleaning, and output times using `time.perf_counter()`, reported in milliseconds.
+- **Cleanup Metrics**: Tracks the number of nulls, duplicates, short entries, and malformed entries removed.
+- **Visualization**: Uses Matplotlib to plot a bar chart (`cleanup_stats.png`):
+  - Bars: Number of entries removed per category (Nulls, Duplicates, Short, Malformed).
+  - Palette: Professional muted colors for enterprise aesthetics.
+## Gradio Interface for Interactive Demo
+The demo is accessible via Gradio, providing an interactive data preprocessing experience:
+- **Input**: Upload a sample call center CSV or use the embedded demo dataset.
+- **Outputs**:
+  - **Cleaned Dataset**: Download `cleaned_call_center_faqs.csv`.
+  - **Cleanup Stats**: Detailed breakdown (e.g., “Cleaned FAQs: 6; removed 4 junk entries: 2 nulls, 1 duplicates, 1 short, 0 malformed”).
+  - **Performance Plot**: Visual metrics for processing time and cleanup stats.
+- **Styling**: Custom dark theme CSS (`#2a2a2a` background, blue buttons) for a sleek, enterprise-ready UI.
+## Setup
+- Clone this repository to a Hugging Face Model repository (free tier, public).
+- Add `requirements.txt` with dependencies (`gradio==4.44.0`, `pandas==2.2.3`, `matplotlib==3.9.2`, etc.).
+- Upload `app.py` (includes embedded demo dataset for seamless deployment).
+- Configure to run with Python 3.9+, CPU hardware (no GPU).
+## Usage
+- **Upload CSV**: Provide a call center CSV in the Gradio UI, or use the default demo dataset.
+- **Output**:
+  - **Cleaned Dataset**: Download the processed `cleaned_call_center_faqs.csv`.
+  - **Cleanup Stats**: “Cleaned FAQs: 6; removed 4 junk entries: 2 nulls, 1 duplicates, 1 short, 0 malformed”.
+  - **Performance Plot**: Visual metrics for processing time and cleanup stats.
+**Example**:
+- **Input CSV**: Sample dataset with 10 FAQs, including 2 nulls, 1 duplicate, 1 short entry.
+- **Output**:
+  - Cleaned Dataset: 6 FAQs in `cleaned_call_center_faqs.csv`.
+  - Cleanup Stats: “Cleaned FAQs: 6; removed 4 junk entries: 2 nulls, 1 duplicates, 1 short, 0 malformed”.
+  - Plot: Processing Time (Ingestion: 50ms, Cleaning: 30ms, Output: 10ms), Cleanup Stats (Nulls: 2, Duplicates: 1, Short: 1, Malformed: 0).
+## Technical Details
+**Stack**:
+- **Pandas**: Data wrangling and preprocessing for call center CSVs.
+- **Gradio**: Interactive UI for real-time data preprocessing demos.
+- **Matplotlib**: Performance visualization with bar charts.
+- **FastAPI Compatibility**: Designed with API-driven preprocessing in mind, leveraging your experience with FastAPI for scalable deployments.
+**Free Tier Optimization**: Lightweight with CPU-only dependencies, no GPU required.
+**Extensibility**: Ready for integration with RAG/CAG pipelines, and cloud deployments on AWS Lambda or Azure Functions.
+## Purpose
+This demo showcases expertise in data preprocessing for AI-driven CX automation, focusing on call center data quality. Built on over 5 years of experience in AI, data engineering, and enterprise-grade deployments, it demonstrates the power of Pandas-based data cleaning for RAG/CAG pipelines, making it ideal for advanced CX solutions in call center environments.
+## Latest Update
+**Status Update**: Placeholder update - January 01, 2025 📝
+- Placeholder update text.
+## Future Enhancements
+- **Real-Time Streaming**: Add support for real-time data streaming from Kafka for live preprocessing.
+- **FastAPI Deployment**: Expose preprocessing pipeline via FastAPI endpoints for production-grade use.
+- **Advanced Validation**: Implement stricter data validation rules using machine learning-based outlier detection.
+- **Cloud Integration**: Enhance compatibility with AWS Glue or Azure Data Factory for enterprise data pipelines.
+**Website**: https://ghostainews.com/
+**Discord**: https://discord.gg/BfA23aYz