Spaces:
Sleeping
Sleeping
| # Codebase Inventory: ml-polymer-recycling | |
| ## Overview | |
| A comprehensive machine learning system for AI-driven polymer aging prediction and classification using spectral data analysis. The project implements multiple CNN architectures (Figure2CNN, ResNet1D, ResNet18Vision) to classify polymer degradation levels as a proxy for recyclability, built with Python, PyTorch, and featuring both CLI and Streamlit UI workflows. | |
| ## Inventory by Category | |
| ### 1. Core Application Modules | |
| - **Module Name**: `models/registry.py` | |
| - **Purpose**: Central registry system for model architectures providing dynamic model selection and instantiation | |
| - **Key Exports/Functions**: `choices()`, `build(name, input_length)`, `_REGISTRY` | |
| - **Key Dependencies**: `models.figure2_cnn`, `models.resnet_cnn`, `models.resnet18_vision` | |
| - **External Dependencies**: `typing` | |
| - **Module Name**: `models/figure2_cnn.py` | |
| - **Purpose**: CNN architecture implementation based on literature (Neo et al. 2023) for 1D Raman spectral classification | |
| - **Key Exports/Functions**: `Figure2CNN` class with conv blocks and classifier layers | |
| - **Key Dependencies**: None (self-contained) | |
| - **External Dependencies**: `torch`, `torch.nn` | |
| - **Module Name**: `models/resnet_cnn.py` | |
| - **Purpose**: ResNet1D implementation with residual blocks for deeper spectral feature learning | |
| - **Key Exports/Functions**: `ResNet1D`, `ResidualBlock1D` classes | |
| - **Key Dependencies**: None (self-contained) | |
| - **External Dependencies**: `torch`, `torch.nn` | |
| - **Module Name**: `models/resnet18_vision.py` | |
| - **Purpose**: ResNet18 architecture adapted for 1D spectral data processing | |
| - **Key Exports/Functions**: `ResNet18Vision` class | |
| - **Key Dependencies**: None (self-contained) | |
| - **External Dependencies**: `torch`, `torch.nn` | |
| - **Module Name**: `utils/preprocessing.py` | |
| - **Purpose**: Spectral data preprocessing utilities including resampling, baseline correction, smoothing, and normalization | |
| - **Key Exports/Functions**: `preprocess_spectrum()`, `resample_spectrum()`, `remove_baseline()`, `normalize_spectrum()`, `smooth_spectrum()` | |
| - **Key Dependencies**: None (self-contained) | |
| - **External Dependencies**: `numpy`, `scipy.interpolate`, `scipy.signal`, `sklearn.preprocessing` | |
| - **Module Name**: `scripts/preprocess_dataset.py` | |
| - **Purpose**: Comprehensive dataset preprocessing pipeline with CLI interface for Raman spectral data | |
| - **Key Exports/Functions**: `preprocess_dataset()`, `resample_spectrum()`, `label_file()`, preprocessing helper functions | |
| - **Key Dependencies**: `scripts.discover_raman_files`, `scripts.plot_spectrum` | |
| - **External Dependencies**: `numpy`, `scipy`, `sklearn.preprocessing` | |
| ### 2. Scripts & Automation | |
| - **Script Name**: `validate_pipeline.sh` | |
| - **Trigger**: Manual execution (`./validate_pipeline.sh`) | |
| - **Apparent Function**: Canonical smoke test validating the complete Raman pipeline from preprocessing through training to inference | |
| - **Dependencies**: `conda`, `scripts/preprocess_dataset.py`, `scripts/train_model.py`, `scripts/run_inference.py`, `scripts/plot_spectrum.py` | |
| - **Script Name**: `scripts/train_model.py` | |
| - **Trigger**: CLI execution (`python scripts/train_model.py`) | |
| - **Apparent Function**: 10-fold stratified cross-validation training with multiple model architectures and preprocessing options | |
| - **Dependencies**: `scripts/preprocess_dataset`, `models/registry`, reproducibility seeds, PyTorch training loop | |
| - **Script Name**: `scripts/run_inference.py` | |
| - **Trigger**: CLI execution (`python scripts/run_inference.py`) | |
| - **Apparent Function**: Single spectrum inference with model loading, preprocessing, and prediction output to JSON | |
| - **Dependencies**: `models/registry`, `scripts/preprocess_dataset`, trained model weights | |
| - **Script Name**: `scripts/plot_spectrum.py` | |
| - **Trigger**: CLI execution (`python scripts/plot_spectrum.py`) | |
| - **Apparent Function**: Visualization tool for Raman spectra with matplotlib plotting and file I/O | |
| - **Dependencies**: Spectrum loading utilities | |
| - **Script Name**: `scripts/discover_raman_files.py` | |
| - **Trigger**: Imported by other scripts | |
| - **Apparent Function**: File discovery and labeling utilities for Raman dataset management | |
| - **Dependencies**: File system operations, regex pattern matching | |
| - **Script Name**: `scripts/list_spectra.py` | |
| - **Trigger**: CLI or import | |
| - **Apparent Function**: Dataset inventory and spectrum listing utilities | |
| - **Dependencies**: File system scanning | |
| ### 3. Configuration & Data | |
| - **File Name**: `deploy/hf-space/requirements.txt` | |
| - **Purpose**: Python dependencies for Hugging Face Spaces deployment | |
| - **Key Contents/Structure**: `streamlit`, `torch`, `torchvision`, `scikit-learn`, `scipy`, `numpy`, `pandas`, `matplotlib`, `fastapi`, `altair`, `huggingface-hub` | |
| - **File Name**: `deploy/hf-space/Dockerfile` | |
| - **Purpose**: Container configuration for Hugging Face Spaces deployment | |
| - **Key Contents/Structure**: Python 3.13-slim base, build tools installation, Streamlit server configuration on port 8501 | |
| - **File Name**: `deploy/hf-space/sample_data/sta-1.txt` | |
| - **Purpose**: Sample Raman spectrum for UI demonstration | |
| - **Key Contents/Structure**: Two-column wavenumber/intensity data format | |
| - **File Name**: `deploy/hf-space/sample_data/sta-2.txt` | |
| - **Purpose**: Additional sample Raman spectrum for UI testing | |
| - **Key Contents/Structure**: Two-column wavenumber/intensity data format | |
| - **File Name**: `.gitignore` | |
| - **Purpose**: Version control exclusions for datasets, build artifacts, and system files | |
| - **Key Contents/Structure**: `datasets/`, `__pycache__/`, model weights, logs, environment files, deprecated scripts | |
| - **File Name**: `MANIFEST.git` | |
| - **Purpose**: Git object manifest listing all tracked files with hashes | |
| - **Key Contents/Structure**: File paths, permissions, and SHA hashes for repository contents | |
| ### 4. Assets & Documentation | |
| - **Asset Name**: `README.md` | |
| - **Purpose**: Primary project documentation with objectives, architecture overview, and usage instructions | |
| - **Key Contents/Structure**: Project goals, model architectures table, structure diagram, installation guides, sample commands | |
| - **Asset Name**: `GROUND_TRUTH_PIPELINE.md` | |
| - **Purpose**: Comprehensive empirical baseline inventory documenting every aspect of the current system | |
| - **Key Contents/Structure**: 635-line detailed documentation of data handling, preprocessing, models, CLI workflow, UI workflow, and gap identification | |
| - **Asset Name**: `docs/ENVIRONMENT_GUIDE.md` | |
| - **Purpose**: Environment management guide for local and HPC deployment | |
| - **Key Contents/Structure**: Conda vs venv setup instructions, platform-specific configurations, dependency management | |
| - **Asset Name**: `docs/PROJECT_TIMELINE.md` | |
| - **Purpose**: Development milestone tracking and project progression documentation | |
| - **Key Contents/Structure**: Phase-based timeline from project kickoff through model expansion, tagged milestones | |
| - **Asset Name**: `docs/sprint_log.md` | |
| - **Purpose**: Sprint-based development log with specific technical changes and testing results | |
| - **Key Contents/Structure**: Chronological entries with goals, changes, tests, and notes for each development sprint | |
| - **Asset Name**: `docs/REPRODUCIBILITY.md` | |
| - **Purpose**: Scientific reproducibility guidelines and artifact control documentation | |
| - **Key Contents/Structure**: Validation procedures, artifact integrity, experimental controls | |
| - **Asset Name**: `docs/HPC_REMOTE_SETUP.md` | |
| - **Purpose**: High-performance computing environment setup for CWRU Pioneer cluster | |
| - **Key Contents/Structure**: HPC-specific configurations, remote access procedures, computational resource management | |
| - **Asset Name**: `docs/BACKEND_MIGRATION_LOG.md` | |
| - **Purpose**: Technical migration documentation for backend architecture changes | |
| - **Key Contents/Structure**: Migration procedures, compatibility notes, system architecture evolution | |
| ### 5. Deployment & UI Components | |
| - **Module Name**: `deploy/hf-space/app.py` | |
| - **Purpose**: Streamlit web application for polymer classification with file upload and model inference | |
| - **Key Exports/Functions**: Streamlit UI components, model loading, preprocessing pipeline, prediction display | |
| - **Key Dependencies**: `models.figure2_cnn`, `models.resnet_cnn`, `utils.preprocessing` (fallback), `scripts.preprocess_dataset` | |
| - **External Dependencies**: `streamlit`, `torch`, `matplotlib`, `PIL`, `numpy` | |
| ### 6. Model Artifacts & Outputs | |
| - **File Name**: `outputs/resnet_model.pth` | |
| - **Purpose**: Trained ResNet1D model weights for Raman spectrum classification | |
| - **Key Contents/Structure**: PyTorch state dictionary with model parameters | |
| ## Workflows & Interactions | |
| - **CLI Training Pipeline**: The main training workflow starts with `scripts/train_model.py` which imports the model registry (`models/registry.py`) to dynamically select architectures (Figure2CNN, ResNet1D, or ResNet18Vision). It uses `scripts/preprocess_dataset.py` to load and preprocess Raman spectra from `datasets/rdwp/`, applying resampling, baseline correction, smoothing, and normalization. The script performs 10-fold stratified cross-validation and saves trained models to `outputs/{model}_model.pth` with diagnostics to `outputs/logs/`. | |
| - **CLI Inference Pipeline**: Running `scripts/run_inference.py` loads a trained model via the registry, processes a single Raman spectrum file through the same preprocessing pipeline, and outputs predictions in JSON format to `outputs/inference/`. | |
| - **UI Workflow**: The Streamlit application (`deploy/hf-space/app.py`) provides a web interface that loads trained models, accepts file uploads or sample data selection, but currently bypasses the full preprocessing pipeline (missing baseline correction, smoothing, and normalization steps) before running inference. | |
| - **Validation Workflow**: The `validate_pipeline.sh` script orchestrates a complete pipeline test by sequentially running preprocessing, training, inference, and plotting scripts to ensure reproducibility and catch regressions. | |
| - **Model Registry System**: All model architectures are centrally managed through `models/registry.py`, which provides dynamic model selection for both CLI training and inference scripts, ensuring consistent model instantiation across the codebase. | |
| ## External Dependencies Summary | |
| - **PyTorch Ecosystem**: `torch`, `torchvision` for deep learning model implementation and training | |
| - **Scientific Computing**: `numpy`, `scipy` for numerical operations and signal processing | |
| - **Machine Learning**: `scikit-learn` for preprocessing, metrics, and cross-validation utilities | |
| - **Data Handling**: `pandas` for structured data manipulation | |
| - **Visualization**: `matplotlib`, `seaborn` for plotting and data visualization | |
| - **Web Framework**: `streamlit` for interactive web application deployment | |
| - **Image Processing**: `PIL` (Pillow) for image handling in the UI | |
| - **Development Tools**: `argparse` for CLI interfaces, `json` for data serialization | |
| - **Deployment**: `fastapi`, `uvicorn` for potential API deployment, `huggingface-hub` for model hosting | |
| ## Key Findings & Assumptions | |
| - **Critical Preprocessing Gap**: The UI workflow in `deploy/hf-space/app.py` bypasses essential preprocessing steps (baseline correction, smoothing, normalization) that are standard in the CLI pipeline, potentially causing prediction inconsistencies. | |
| - **Model Architecture Assumptions**: Three CNN architectures are registered (`figure2`, `resnet`, `resnet18vision`) but the codebase suggests only two are currently trained and validated in the standard pipeline. | |
| - **Dataset Structure**: The system assumes Raman spectra are stored as two-column text files (wavenumber, intensity) in the `datasets/rdwp/` directory, with filenames indicating weathering conditions for automated labeling. | |
| - **Environment Fragmentation**: The project uses different dependency management systems (Conda for local development, venv for HPC, pip requirements for deployment) which could lead to environment inconsistencies. | |
| - **Reproducibility Controls**: Strong emphasis on scientific reproducibility with fixed random seeds, deterministic algorithms, and comprehensive validation scripts, indicating this is research-oriented code requiring strict experimental controls. | |
| - **Deployment Readiness**: The Hugging Face Spaces deployment setup suggests the project is intended for public demonstration or research sharing, but the preprocessing gap needs resolution for production use. | |
| - **Legacy Code Management**: The `.gitignore` and documentation references suggest active management of deprecated FTIR-related components, indicating focused scope refinement to Raman-only analysis. | |