crossword-app/backend-py/README.md

# Python Backend with Vector Similarity Search

This is the Python implementation of the crossword generator backend, featuring true AI word generation via vector similarity search.

## 🚀 Features

- **True Vector Search**: Uses sentence-transformers + FAISS for semantic word discovery
- **30K+ Vocabulary**: Searches through full model vocabulary instead of limited static lists
- **FastAPI**: Modern, fast Python web framework
- **Same API**: Compatible with existing React frontend
- **Hybrid Approach**: AI vector search with static word fallback

## 🔄 Differences from JavaScript Backend

| Feature | JavaScript Backend | Python Backend |
|---------|-------------------|----------------|
| **Word Generation** | Embedding filtering of static lists | True vector similarity search |
| **Vocabulary Size** | ~100 words per topic | 30K+ words from model |
| **AI Approach** | Semantic similarity filtering | Nearest neighbor search |
| **Performance** | Fast but limited | Slower startup, better results |
| **Dependencies** | Node.js + HuggingFace API | Python + ML libraries |

## 🛠️ Setup & Installation

### Prerequisites
- Python 3.11+ (3.11 recommended for Docker compatibility)
- pip (Python package manager)

### Basic Setup (Core Functionality)
```bash
# Clone and navigate to backend directory
cd crossword-app/backend-py

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install core dependencies
pip install -r requirements.txt

# Start the server
python app.py
```

### Full Development Setup (with AI features)
```bash
# Install development dependencies including AI/ML libraries
pip install -r requirements-dev.txt

# This includes:
# - All core dependencies
# - AI/ML libraries (torch, sentence-transformers, etc.)
# - Development tools (pytest, coverage, etc.)
```

### Requirements Files
- **`requirements.txt`**: Core dependencies for basic functionality
- **`requirements-dev.txt`**: Full development environment with AI features

> **Note**: The AI/ML dependencies are large (~2GB). For basic testing without AI features, use `requirements.txt` only.

> **Python Version**: Both local development and Docker use Python 3.11+ for optimal performance and latest package compatibility.

## 📁 Structure

```
backend-py/
├── app.py                          # FastAPI application entry point
├── requirements.txt                # Core Python dependencies
├── requirements-dev.txt            # Full development dependencies
├── src/
│   ├── services/
│   │   ├── vector_search.py        # Core vector similarity search
│   │   └── crossword_generator.py  # Puzzle generation logic
│   └── routes/
│       └── api.py                  # API endpoints (matches JS backend)
├── test-unit/                      # Unit tests (pytest framework) - 5 files
│   ├── test_crossword_generator.py
│   ├── test_api_routes.py
│   └── test_vector_search.py
├── test-integration/               # Integration tests (standalone scripts) - 16 files
│   ├── test_simple_generation.py
│   ├── test_boundary_fix.py
│   └── test_local.py               # (+ 13 more test files)
├── data/ -> ../backend/data/       # Symlink to shared word data
└── public/                         # Frontend static files (copied during build)
```

## 🛠 Dependencies

### Core ML Stack
- `sentence-transformers`: Local model loading and embeddings
- `faiss-cpu`: Fast vector similarity search
- `torch`: PyTorch for model inference
- `numpy`: Vector operations

### Web Framework
- `fastapi`: Modern Python web framework
- `uvicorn`: ASGI server
- `pydantic`: Data validation

### Testing
- `pytest`: Testing framework
- `pytest-asyncio`: Async test support

## 🧪 Testing

### 📁 Test Organization (Reorganized for Clarity)

**We've reorganized the test structure for better developer experience:**

| Test Type | Location | Purpose | Framework | Count |
|-----------|----------|---------|-----------|-------|
| **Unit Tests** | `test-unit/` | Test individual components in isolation | pytest | 5 files |
| **Integration Tests** | `test-integration/` | Test complete workflows end-to-end | Standalone scripts | 16 files |

**Benefits of this structure:**
- ✅ **Clear separation** between unit and integration testing
- ✅ **Intuitive naming** - developers immediately understand test types
- ✅ **Better tooling** - can run different test types independently
- ✅ **Easier maintenance** - organized by testing strategy

> **Note**: Previously tests were mixed in `tests/` folder and root-level `test_*.py` files. The new structure provides much better organization.

### Unit Tests Details (`test-unit/`)

**What they test:** Individual components with mocking and isolation
- `test_crossword_generator.py` - Core crossword generation logic
- `test_api_routes.py` - FastAPI endpoint handlers  
- `test_crossword_generator_wrapper.py` - Service wrapper layer
- `test_index_bug_fix.py` - Specific bug fix validations
- `test_vector_search.py` - AI vector search functionality (requires torch)

### Run Unit Tests (Formal Test Suite)
```bash
# Run all unit tests
python run_tests.py

# Run specific test modules  
python run_tests.py crossword_generator
pytest test-unit/test_crossword_generator.py -v

# Run core tests (excluding AI dependencies)
pytest test-unit/ -v --ignore=test-unit/test_vector_search.py

# Run individual unit test classes
pytest test-unit/test_crossword_generator.py::TestCrosswordGenerator::test_init -v
```

### Integration Tests Details (`test-integration/`)

**What they test:** Complete workflows without mocking - real functionality
- `test_simple_generation.py` - End-to-end crossword generation
- `test_boundary_fix.py` - Word boundary validation (our major fix!)
- `test_local.py` - Local environment and dependencies
- `test_word_boundaries.py` - Comprehensive boundary testing
- `test_bounds_comprehensive.py` - Advanced bounds checking
- `test_final_validation.py` - API integration testing
- And 10 more specialized feature tests...

### Run Integration Tests (End-to-End Scripts)
```bash
# Test core functionality
python test-integration/test_simple_generation.py
python test-integration/test_boundary_fix.py
python test-integration/test_local.py

# Test specific features
python test-integration/test_word_boundaries.py
python test-integration/test_bounds_comprehensive.py

# Test API integration
python test-integration/test_final_validation.py
```

### Test Coverage
```bash
# Run core tests with coverage (requires requirements-dev.txt)
pytest test-unit/test_crossword_generator.py --cov=src --cov-report=html
pytest test-unit/test_crossword_generator.py --cov=src --cov-report=term

# Full coverage report (may fail without AI dependencies)
pytest test-unit/ --cov=src --cov-report=html --ignore=test-unit/test_vector_search.py
```

### Test Status
- ✅ **Core crossword generation**: 15/19 unit tests passing
- ✅ **Boundary validation**: All integration tests passing
- ⚠️ **AI/Vector search**: Requires torch dependencies
- ⚠️ **Some async mocking**: Minor test infrastructure issues

### 🔄 Migration Guide (For Existing Developers)

**If you had previous commands, update them:**

| Old Command | New Command |
|-------------|-------------|
| `pytest tests/` | `pytest test-unit/` |
| `python test_simple_generation.py` | `python test-integration/test_simple_generation.py` |
| `pytest tests/ --cov=src` | `pytest test-unit/ --cov=src` |

**All functionality is preserved** - just organized better!

## 🔧 Configuration

Environment variables (set in HuggingFace Spaces):

```bash
# Core settings
PORT=7860
NODE_ENV=production

# AI Configuration
EMBEDDING_MODEL=sentence-transformers/all-mpnet-base-v2
WORD_SIMILARITY_THRESHOLD=0.65

# Optional
LOG_LEVEL=INFO
```

## 🎯 Vector Search Process

1. **Initialization**:
   - Load sentence-transformers model locally
   - Extract 30K+ vocabulary from model tokenizer
   - Pre-compute embeddings for all vocabulary words
   - Build FAISS index for fast similarity search

2. **Word Generation**:
   - Get topic embedding: `"Animals" → [768-dim vector]`
   - Search FAISS index for nearest neighbors
   - Filter by similarity threshold (0.65+)
   - Filter by difficulty (word length)
   - Return top matches with generated clues

3. **Fallback**:
   - If vector search fails → use static word lists
   - If insufficient AI words → supplement with static words

## 🧪 Testing

```bash
# Local testing (without full vector search)
cd backend-py
python test_local.py

# Start development server
python app.py
```

## 🐳 Docker Deployment

The Dockerfile has been updated to use Python backend:

```dockerfile
FROM python:3.9-slim
# ... install dependencies
# ... build frontend (same as before)
# ... copy to backend-py/public/
CMD ["python", "app.py"]
```

## 🧪 Testing

### Quick Test
```bash
# Basic functionality test (no model download)
python test_local.py
```

### Comprehensive Unit Tests
```bash
# Run all unit tests
python run_tests.py

# Or use pytest directly
pytest tests/ -v

# Run specific test file
python run_tests.py crossword_generator_fixed
pytest tests/test_crossword_generator_fixed.py -v

# Run with coverage
pytest tests/ --cov=src --cov-report=html
```

### Test Structure
- `tests/test_crossword_generator_fixed.py` - Core grid generation logic
- `tests/test_vector_search.py` - Vector similarity search 
- `tests/test_crossword_generator_wrapper.py` - Service wrapper
- `tests/test_api_routes.py` - FastAPI endpoints

### Key Test Features
- ✅ **Index alignment fix**: Tests the list index out of range bug fix
- ✅ **Mocked vector search**: Tests without downloading models
- ✅ **API validation**: Tests all endpoints and error cases
- ✅ **Async support**: Full pytest-asyncio integration
- ✅ **Error handling**: Tests malformed inputs and edge cases

## 📊 Performance Comparison

**Startup Time**:
- JavaScript: ~2 seconds
- Python: ~30-60 seconds (model download + index building)

**Word Quality**:
- JavaScript: Limited by static word lists
- Python: Access to full model vocabulary with semantic understanding

**Memory Usage**:
- JavaScript: ~100MB
- Python: ~500MB-1GB (model + embeddings + FAISS index)

**API Response Time**:
- JavaScript: ~100ms (after cache warm-up)
- Python: ~200-500ms (vector search + filtering)

## 🔄 Migration Strategy

1. **Phase 1** ✅: Basic Python backend structure
2. **Phase 2**: Test vector search functionality  
3. **Phase 3**: Docker deployment and production testing
4. **Phase 4**: Compare with JavaScript backend
5. **Phase 5**: Production switch with rollback capability

## 🎯 Next Steps

- [ ] Test vector search with real model
- [ ] Optimize FAISS index performance
- [ ] Add more sophisticated crossword grid generation
- [ ] Implement LLM-based clue generation
- [ ] Add caching for frequently requested topics