docs: add soft minimum visualization ideas and vocabulary alternatives analysis

- Add comprehensive visualization concepts for soft minimum method
- Add detailed analysis of vocabulary alternatives beyond WordFreq
- Add python scripts to analyse word list from peter norvig home page
- Update .gitignore for fine-tuned models and T5 model cache
- Include SUBTLEX dataset and Norvig vocabulary analysis files

Signed-off-by: Vimal Kumar <[email protected]>

crossword-app/backend-py/docs/softmin_visualization_ideas.md

@@ -0,0 +1,227 @@
+# Soft Minimum Visualization Ideas
+
+This document outlines visualization concepts to showcase how the soft minimum method works for multi-topic word intersection in the crossword generator.
+
+## Overview
+
+The soft minimum method uses the formula `-log(sum(exp(-beta * similarities))) / beta` to find words that are genuinely relevant to ALL topics simultaneously. Unlike simple averaging, which can promote words that are highly relevant to just one topic, soft minimum penalizes words that score poorly on any individual topic.
+
+Visualizations would help users understand:
+- How soft minimum differs from averaging
+- Why it produces better semantic intersections
+- How the beta parameter affects results
+- How adaptive beta mechanism works
+
+## Visualization Concepts
+
+### 1. Heat Map Comparison (🌟 Most Impactful)
+
+**Concept**: Side-by-side heat maps showing individual topic similarities vs soft minimum scores.
+
+**Layout**:
+- **Left Heat Map**: Individual Similarities
+  - Rows: Top 50-100 words
+  - Columns: Individual topics (e.g., "universe", "movies", "languages")
+  - Color intensity: Similarity score (0.0 = white, 1.0 = dark blue)
+  
+- **Right Heat Map**: Soft Minimum Results
+  - Same rows (words)
+  - Single column: Soft minimum score
+  - Color intensity: Final soft minimum score
+
+**Key Insights**:
+- Words like "anime" would show moderate blue across all topics → high soft minimum score
+- Words like "astronomy" would show dark blue for "universe", white for others → low soft minimum score
+- Visually demonstrates how soft minimum penalizes topic-specific words
+
+**Implementation**: 
+- Frontend: Use libraries like D3.js or Plotly for interactive heat maps
+- Backend: Return individual topic similarities alongside soft minimum scores
+
+### 2. 3D Scatter Plot (For 3-Topic Cases)
+
+**Concept**: 3D space where each axis represents similarity to one topic.
+
+**Layout**:
+- X-axis: Similarity to topic 1
+- Y-axis: Similarity to topic 2
+- Z-axis: Similarity to topic 3
+- Point size/color: Soft minimum score
+- Point labels: Word names (on hover)
+
+**Key Insights**:
+- Words near the center (similar to all topics) = large, bright points
+- Words near axes (similar to only one topic) = small, dim points
+- Shows the "volume" of intersection vs union
+
+**Implementation**:
+- Use Three.js or Plotly 3D
+- Interactive rotation and zoom
+- Filter points by soft minimum threshold
+
+### 3. Interactive Beta Slider
+
+**Concept**: Real-time visualization of how beta parameter affects word selection.
+
+**Layout**:
+- Horizontal slider: Beta value (1.0 to 20.0)
+- Bar chart: Word scores (sorted descending)
+- Threshold line: Current similarity threshold
+- Counter: Number of words above threshold
+
+**Key Insights**:
+- High beta (strict): Only a few words pass, distribution is peaked
+- Low beta (permissive): More words pass, distribution flattens
+- Shows adaptive beta mechanism in action
+
+**Implementation**:
+- React component with range slider
+- Real-time recalculation of soft minimum scores
+- Animated transitions as beta changes
+
+### 4. Venn Diagram with Words
+
+**Concept**: Position words in Venn diagram based on topic similarities.
+
+**Layout** (for 2-3 topics):
+- Circles represent individual topics
+- Words positioned based on similarity combinations
+- Words in intersections = high soft minimum scores
+- Words in single circles = low soft minimum scores
+- Word opacity/size based on final soft minimum score
+
+**Key Insights**:
+- Visual representation of "true intersections"
+- Words in overlap regions are what soft minimum promotes
+- Empty intersection regions explain why some topic combinations yield few words
+
+**Implementation**:
+- SVG-based Venn diagrams
+- Dynamic positioning algorithm
+- Interactive word tooltips
+
+### 5. Before/After Word Clouds
+
+**Concept**: Compare averaging vs soft minimum results using word clouds.
+
+**Layout**:
+- **Left Cloud**: "Averaging Method"
+  - Word size based on average similarity
+  - May prominently feature problematic words like "ethology" for Art+Books
+  
+- **Right Cloud**: "Soft Minimum Method"
+  - Word size based on soft minimum score
+  - Should prominently feature true intersections like "literature"
+
+**Key Insights**:
+- Dramatic visual difference in word prominence
+- Shows quality improvement at a glance
+- Easy to understand for non-technical users
+
+**Implementation**:
+- Use word cloud libraries (wordcloud2.js, D3-cloud)
+- Color coding by topic affinity
+- Interactive word selection
+
+### 6. Mathematical Formula Animation
+
+**Concept**: Step-by-step visualization of soft minimum calculation.
+
+**Layout**:
+- Example word with similarities: [0.8, 0.2, 0.1] (universe, movies, languages)
+- Animated steps:
+  1. Show individual similarities as bars
+  2. Apply exponential transformation: exp(-beta * sim)
+  3. Sum the exponentials
+  4. Apply logarithm and normalization
+  5. Compare result to simple average (0.37)
+
+**Key Insights**:
+- How the minimum similarity dominates the calculation
+- Why soft minimum ≈ minimum similarity for high beta
+- Mathematical intuition behind the formula
+
+**Implementation**:
+- Animated SVG or Canvas
+- Step-by-step button progression
+- Mathematical notation display
+
+### 7. Adaptive Beta Journey
+
+**Concept**: Show the adaptive beta retry process as a timeline.
+
+**Layout**:
+- Horizontal timeline showing beta decay: 10.0 → 7.0 → 4.9 → 3.4...
+- For each beta value:
+  - Histogram of soft minimum scores
+  - Threshold line (adjusted)
+  - Count of valid words
+  - Decision: "Continue" or "Stop"
+
+**Key Insights**:
+- How threshold adjustment makes lower beta more permissive
+- Why word count increases with each retry
+- When the algorithm decides to stop
+
+**Implementation**:
+- Timeline component with expandable sections
+- Small multiples showing score distributions
+- Real-time data from debug logs
+
+## Implementation Priorities
+
+### Phase 1: Essential (MVP)
+1. **Heat Map Comparison** - Most educational value
+2. **Interactive Beta Slider** - Shows parameter effects clearly
+
+### Phase 2: Enhanced Understanding
+3. **Before/After Word Clouds** - Easy to understand impact
+4. **Mathematical Formula Animation** - Educational for technical users
+
+### Phase 3: Advanced Analysis
+5. **3D Scatter Plot** - For deep analysis of 3-topic cases
+6. **Venn Diagram** - Complex positioning algorithms
+7. **Adaptive Beta Journey** - Comprehensive debugging tool
+
+## Technical Implementation Notes
+
+### Backend Changes Needed
+- Return individual topic similarities alongside soft minimum scores
+- Add debug endpoint for visualization data
+- Include beta parameter and threshold information in responses
+
+### Frontend Integration
+- Add to existing debug tab
+- Use React components for interactivity
+- Responsive design for different screen sizes
+- Export/save visualization capabilities
+
+### Data Format
+```json
+{
+  "visualization_data": {
+    "individual_similarities": {
+      "word1": [0.8, 0.2, 0.1],
+      "word2": [0.3, 0.9, 0.4]
+    },
+    "soft_minimum_scores": {
+      "word1": 0.15,
+      "word2": 0.32
+    },
+    "beta_used": 7.0,
+    "threshold_adjusted": 0.175,
+    "topics": ["universe", "movies", "languages"]
+  }
+}
+```
+
+## Expected Impact
+
+These visualizations would:
+1. **Educate users** about the soft minimum method
+2. **Build confidence** in the algorithm's choices
+3. **Enable debugging** of problematic topic combinations
+4. **Facilitate research** into parameter optimization
+5. **Demonstrate value** of the multi-topic intersection approach
+
+The heat map comparison alone would be worth implementing, as it clearly shows why soft minimum produces higher-quality word intersections than simple averaging.

crossword-app/backend-py/docs/vocabulary_alternatives_analysis.md

@@ -0,0 +1,405 @@
+# Vocabulary Alternatives Analysis: Beyond WordFreq
+
+## Executive Summary
+
+WordFreq, while useful for general frequency analysis, produces vocabulary quality issues for crossword generation due to its web-scraped, uncurated nature. After hands-on evaluation of alternatives, most "curated" crossword lists have significant quality issues requiring substantial cleanup effort.
+
+### **Updated Recommendations (Post-Evaluation):**
+1. **Primary**: COCA free sample (6K high-quality words with rich metadata) + Peter Norvig's clean 100K list
+2. **Quality Leader**: COCA full version (if budget allows) - 14 billion words, sophisticated metadata
+3. **Fallback**: SUBTLEX (reasonable quality, needs programming to parse properly)  
+4. **Avoid**: Most crossword-specific lists contain junk data requiring extensive cleanup
+5. **Semantic Processing**: Keep all-mpnet-base-v2 (working well)
+
+## Current Issues with WordFreq Vocabulary
+
+### Problems Identified:
+1. **Web-based contamination**: Includes Reddit, Twitter, and web crawl data with typos, slang, and internet-specific language
+2. **No quality filtering**: Purely frequency-based without considering appropriateness for crosswords
+3. **Mixed registers**: Combines formal and informal language indiscriminately  
+4. **Problematic intersections**: Generates words like "ethology", "guns", "porn" for topics like "Art+Books"
+5. **Limited metadata**: No information about word suitability, part-of-speech, or crossword usage
+6. **AI contamination risk**: WordFreq author stopped updates in 2024 due to generative AI polluting data sources
+
+### Impact on Crossword Generation:
+- Lower quality semantic intersections
+- Inappropriate words for family-friendly puzzles
+- Poor difficulty calibration
+- Reduced solver experience quality
+
+## Superior Alternatives
+
+### 1. Crossword-Specific Word Lists (⚠️ QUALITY ISSUES FOUND)
+
+#### A. Collaborative Word List (❌ NOT RECOMMENDED)
+- **Source**: https://github.com/Crossword-Nexus/collaborative-word-list
+- **Size**: 114,000+ words
+- **Direct download**: `https://raw.githubusercontent.com/Crossword-Nexus/collaborative-word-list/main/xwordlist.dict`
+- **QUALITY PROBLEMS IDENTIFIED**:
+  - Contains nonsensical entries: `10THGENCONSOLE`, `1STGENERATIONCONSOLES`, `4XGAMES`
+  - Single letters: `A`, `AA`, `AAA`, `AAAA`
+  - Meaningless sequences: `AAAAH`, `AAAAUTOCLUB`
+  - **Verdict**: Requires extensive cleanup before use
+
+#### B. Spread the Word(list) (❌ NOT RECOMMENDED)
+- **Source**: https://www.spreadthewordlist.com
+- **Size**: 114,000+ answers with scores
+- **QUALITY PROBLEMS IDENTIFIED**:
+  - Garbage entries: `zzzzzzzzzzzzzzz`, `zzzquil`
+  - Malformed words: `aaaaddress`, `aabb`, `aabba`
+  - Random sequences: `aaiiiiiiiiiiiii`
+  - **Verdict**: Same quality issues as Collaborative List
+
+#### C. Christopher Jones' Crossword Wordlist (⚠️ NEEDS CLEANUP)
+- **Source**: https://github.com/christophsjones/crossword-wordlist
+- **QUALITY PROBLEMS IDENTIFIED**:
+  - Long phrases: `"a week from now"`, `"a recipe for disaster"`
+  - Absurdly long compounds: `ABIRDINTHEHANDISWORTHTWOINTHEBUSH`, `ABLEBODIEDSEAMAN`
+  - Arbitrary scoring: Many words with score 50 don't match claimed "common words you wouldn't hesitate to use"
+  - **Verdict**: Contains good data but needs significant filtering and rescoring
+
+### 2. SUBTLEX Psycholinguistic Databases (✅ REASONABLE QUALITY)
+
+#### SUBTLEX-US (American English)
+- **Source**: https://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus
+- **Size**: 74,000+ words
+- **Quality**: Based on film/TV subtitles (natural language exposure)
+- **Scoring**: Zipf scale 1-7, contextual diversity metrics
+- **License**: Free for research
+
+#### EVALUATION RESULTS:
+- **✅ Better quality**: Words are generally reasonable and appropriate
+- **⚠️ Contains tiny phrases**: Some entries are short phrases rather than single words
+- **⚠️ Requires programming**: Need to parse and filter the numerical data properly
+- **✅ Rich metadata**: Includes frequency, Zipf scores, part-of-speech, contextual diversity
+- **✅ Research backing**: Proven to predict word processing difficulty better than traditional corpora
+
+#### Advantages:
+- **Psycholinguistic validity**: Better predictor of word processing difficulty
+- **Clean vocabulary**: Professional media content (edited, appropriate)  
+- **Good difficulty calibration**: Zipf 1-3 = rare/hard, 4-7 = common/easy
+- **Multiple languages**: Available for US, UK, Chinese, Welsh, Spanish
+
+### 3. COCA (Corpus of Contemporary American English) (🌟 EXCELLENT QUALITY)
+
+#### Available Data:
+- **Free tier**: ~6,000 words with rich metadata and collocates
+- **Full version**: 14 billion words with sophisticated metadata (paid)
+- **Source**: https://www.wordfrequency.info/ and https://github.com/brucewlee/COCA-WordFrequency
+- **Composition**: Balanced across news, fiction, academic, spoken
+
+#### EVALUATION RESULTS:
+- **🌟 Excellent quality**: "Phew, this is good" - professional curation shows
+- **✅ Rich metadata**: Frequency, part-of-speech, genre distribution, collocates
+- **✅ Clean vocabulary**: Academic standard filtering
+- **✅ Balanced representation**: Multiple text types ensure comprehensive coverage
+- **💰 Premium option**: Full version provides 14 billion words with sophisticated metadata
+- **✅ Free sample sufficient**: 6K words could serve as high-quality core vocabulary
+
+#### Advantages:
+- **Academic gold standard**: Most accurate and reliable word frequency data
+- **Professional curation**: High editorial and scholarly standards
+- **Balanced corpus**: News, fiction, academic, spoken genres represented
+- **Collocate data**: Helps understand word usage patterns and context
+- **Research proven**: Widely used and validated in linguistics research
+
+### 4. Peter Norvig's Clean Word Lists (🌟 EXCELLENT DISCOVERY)
+
+#### Norvig's Word Count Lists
+- **Source**: https://norvig.com/ngrams/
+- **Key Resource**: `count_1w100k.txt` - 100,000 most popular words, all uppercase
+- **Quality**: Really clean vocabulary without junk entries
+- **Problem**: No frequency information included
+
+#### EVALUATION RESULTS:
+- **✅ Very clean**: Properly curated, no garbage like other sources
+- **✅ Good coverage**: 100K words should provide sufficient vocabulary
+- **✅ Reliable source**: Peter Norvig (Google's Director of Research) ensures quality
+- **❌ Missing frequencies**: Would need to cross-reference with other sources for difficulty grading
+- **💡 Hybrid opportunity**: Could combine Norvig's clean words with frequency data from SUBTLEX or COCA
+
+#### Potential Implementation:
+```python
+# Use Norvig's clean word list as vocabulary base
+norvig_words = load_norvig_100k()
+# Cross-reference with SUBTLEX for frequency data  
+subtlex_freq = load_subtlex_frequencies()
+# Result: Clean vocabulary + reliable frequency information
+```
+
+### 5. Premium Options (For Comparison - Not Evaluated)
+
+#### XWordInfo (NYT-focused)
+- **Cost**: $50 Angel membership
+- **Quality**: Every NYT crossword ever published  
+- **Size**: 200,000+ words
+- **Note**: Not evaluated in this analysis
+
+#### Cruciverb
+- **Cost**: $35 Gold membership
+- **Quality**: Multiple publication sources
+- **Note**: Not evaluated in this analysis
+
+## Detailed Comparison Analysis (Updated with Evaluation Results)
+
+| Source | Size | Quality Score | Frequency Data | Evaluated Quality | Cost | Recommendation |
+|--------|------|---------------|----------------|------------------|------|----------------|
+| **WordFreq** | 100K+ | ❌ Web-scraped | ✅ Frequency | ❌ Original issues | Free | ⚠️ Current baseline |
+| **Collaborative List** | 114K+ | ❌ Junk entries | ❌ Arbitrary scoring | ❌ `10THGENCONSOLE`, `AAAA` | Free | ❌ **AVOID** |
+| **Spread Wordlist** | 114K+ | ❌ Junk entries | ❌ Arbitrary scoring | ❌ `zzzzzzzzzzzzzzz`, `aabb` | Free | ❌ **AVOID** |
+| **C. Jones Wordlist** | ~50K | ⚠️ Needs filtering | ⚠️ Arbitrary scoring | ⚠️ Long phrases, compounds | Free | ⚠️ **CLEANUP REQUIRED** |
+| **SUBTLEX-US** | 74K | ✅ Reasonable quality | ✅ Zipf 1-7 | ✅ Clean, some phrases | Free | ✅ **VIABLE** |
+| **COCA (free)** | 6K | 🌟 Excellent | ✅ Rich metadata | 🌟 "Phew, this is good" | Free | 🌟 **RECOMMENDED** |
+| **COCA (full)** | 1M+ | 🌟 Excellent | ✅ Rich metadata | 🌟 Sophisticated metadata | $$$ | 🌟 **PREMIUM CHOICE** |
+| **Norvig 100K** | 100K | 🌟 Very clean | ❌ None included | 🌟 Clean, no garbage | Free | 🌟 **HYBRID BASE** |
+
+## Updated Implementation Recommendations (Post-Evaluation)
+
+### Recommended Approach: Hybrid COCA + Norvig System
+
+Based on hands-on evaluation, the cleanest approach combines the best of multiple sources:
+
+#### Option A: COCA Free + Extended Coverage (Recommended)
+```python
+# 1. Load COCA 6K words as high-quality core
+def load_coca_core():
+    """Load 6K high-quality words from COCA free sample"""
+    # Excellent quality, rich metadata, reliable frequencies
+    return parse_coca_free_sample()
+
+# 2. Extend with filtered SUBTLEX for broader coverage  
+def extend_with_subtlex():
+    """Add clean words from SUBTLEX for broader coverage"""
+    # Filter out phrases, keep single words only
+    # Use Zipf scores for difficulty grading
+    return filtered_subtlex_words()
+
+# 3. Cross-reference with Norvig's clean list for validation
+def validate_with_norvig():
+    """Use Norvig's 100K list to validate word cleanliness"""
+    norvig_clean = load_norvig_100k()
+    # Only include words that appear in Norvig's curated list
+    return validated_vocabulary
+```
+
+#### Option B: Norvig Base + Frequency Cross-Reference (Alternative)
+```python
+# 1. Start with Norvig's clean 100K vocabulary
+norvig_words = load_norvig_100k()
+
+# 2. Cross-reference with COCA for frequency data
+coca_freq = load_coca_frequencies()  # Free 6K sample
+subtlex_freq = load_subtlex_frequencies()  # Broader coverage
+
+# 3. Assign frequencies with fallback chain
+def get_word_difficulty(word):
+    if word in coca_freq:
+        return coca_freq[word]  # Highest quality
+    elif word in subtlex_freq:
+        return subtlex_freq[word]  # Good quality
+    else:
+        return default_difficulty  # Fallback
+```
+
+### Why This Hybrid Approach Works
+
+#### Problems with "Crossword-Specific" Lists:
+- **Collaborative Word List**: Contains `10THGENCONSOLE`, `AAAA`, `AAAAUTOCLUB`
+- **Spread the Wordlist**: Contains `zzzzzzzzzzzzzzz`, `aaaaddress`, `aabba`  
+- **Christopher Jones**: Contains `ABIRDINTHEHANDISWORTHTWOINTHEBUSH`
+- **Verdict**: All require extensive cleanup, defeating their supposed advantage
+
+#### Advantages of COCA + Norvig Hybrid:
+- **COCA Free**: 6K professionally curated, academically validated words
+- **Norvig 100K**: Clean vocabulary from Google's Director of Research
+- **SUBTLEX**: Reasonable quality with psycholinguistic validity
+- **No garbage**: Avoid the cleanup nightmare of "crossword-specific" lists
+- **Research backing**: Academic and industry validation
+
+### Updated Difficulty Grading System
+
+```python
+def classify_word_difficulty(word):
+    """Updated difficulty classification using clean sources"""
+    
+    # Priority 1: COCA data (highest quality)
+    if word in coca_frequencies:
+        freq_rank = coca_frequencies[word]['rank']
+        if freq_rank <= 1000:
+            return "easy"
+        elif freq_rank <= 3000:
+            return "medium" 
+        else:
+            return "hard"
+    
+    # Priority 2: SUBTLEX Zipf score
+    elif word in subtlex_zipf:
+        zipf = subtlex_zipf[word]
+        if zipf >= 4.5:
+            return "easy"      # Very common
+        elif zipf >= 2.5:
+            return "medium"    # Moderately common  
+        else:
+            return "hard"      # Rare
+    
+    # Fallback: Conservative classification
+    else:
+        return "medium"  # Unknown words default to medium
+```
+
+## Updated Technical Integration Steps
+
+### 1. Data Download and Preprocessing (Revised)
+
+```bash
+# Download COCA free sample (6K high-quality words)
+wget https://raw.githubusercontent.com/brucewlee/COCA-WordFrequency/master/coca_5000.txt
+
+# Download Peter Norvig's clean 100K word list
+wget https://norvig.com/ngrams/count_1w100k.txt
+
+# Download SUBTLEX-US (requires academic access)
+# Available at: https://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus
+
+# AVOID these due to quality issues:
+# ❌ Collaborative Word List (contains garbage)
+# ❌ Spread the Wordlist (contains garbage) 
+# ❌ Christopher Jones (needs extensive cleanup)
+```
+
+### 2. Data Structure Migration
+
+```python
+class EnhancedVocabulary:
+    def __init__(self):
+        self.collaborative_scores = {}  # word -> quality score (10-100)
+        self.subtlex_zipf = {}         # word -> zipf score (1-7)  
+        self.subtlex_pos = {}          # word -> part of speech
+        self.word_embeddings = {}      # word -> embedding vector
+    
+    def load_all_sources(self):
+        """Load and integrate all vocabulary sources"""
+        self.load_collaborative_wordlist()
+        self.load_subtlex_data()
+        self.compute_embeddings()  # Keep existing all-mpnet-base-v2
+    
+    def is_crossword_suitable(self, word):
+        """Filter based on crossword appropriateness"""
+        return word.upper() in self.collaborative_scores
+```
+
+### 3. Configuration Updates
+
+```python
+# Environment variables to add
+VOCAB_SOURCE = "collaborative"  # "collaborative", "subtlex", "hybrid"
+COLLABORATIVE_WORDLIST_URL = "https://raw.githubusercontent.com/..."
+SUBTLEX_DATA_PATH = "/path/to/subtlex_us.txt"
+MIN_CROSSWORD_QUALITY = 30  # Minimum collaborative score
+MIN_ZIPF_SCORE = 2.0       # Minimum SUBTLEX frequency
+```
+
+## Quality Scoring Systems Comparison
+
+### WordFreq (Current)
+- **Scale**: Frequency values (logarithmic)
+- **Basis**: Web text frequency
+- **Issues**: No quality filtering, includes inappropriate content
+
+### Collaborative Word List
+- **Scale**: 10-100 quality score
+- **Basis**: Crossword constructor consensus
+- **Interpretation**: 
+  - 70-100: Excellent crossword words (common, clean)
+  - 40-69: Good crossword words (moderate difficulty)
+  - 10-39: Challenging words (obscure, specialized)
+
+### SUBTLEX Zipf Scale
+- **Scale**: 1-7 (logarithmic)
+- **Basis**: Psycholinguistic word processing research
+- **Interpretation**:
+  - 6-7: Ultra common (THE, AND, OF)
+  - 4-5: Common (HOUSE, WATER, FRIEND)
+  - 2-3: Uncommon (BIZARRE, ELOQUENT)
+  - 1: Rare (OBSEQUIOUS, PERSPICACIOUS)
+
+## Expected Benefits
+
+### Immediate Quality Improvements:
+1. **Cleaner intersections**: No more "ethology/guns/porn" issues
+2. **Family-friendly vocabulary**: Community-curated appropriateness
+3. **Better difficulty calibration**: Psycholinguistically validated scales
+4. **Crossword-optimized**: Words chosen for puzzle suitability
+
+### Long-term Advantages:
+1. **Community support**: Active maintenance by crossword constructors
+2. **Research backing**: SUBTLEX has extensive academic validation
+3. **Hybrid flexibility**: Can combine multiple quality signals
+4. **Scalability**: Easy to add new vocabulary sources
+
+## Migration Strategy
+
+### Week 1: Data Integration
+- Download and preprocess Collaborative Word List
+- Create vocabulary loading pipeline
+- Implement basic quality filtering
+
+### Week 2: Scoring System
+- Implement hybrid quality scoring
+- Map quality scores to difficulty levels
+- Test with existing multi-topic intersection methods
+
+### Week 3: Performance Validation
+- A/B test against WordFreq baseline
+- Measure semantic intersection quality
+- Validate difficulty calibration
+
+### Week 4: Production Deployment
+- Update environment configuration
+- Monitor vocabulary coverage
+- Collect user feedback on word quality
+
+## Alternative Implementation: Gradual Migration
+
+For lower risk, implement gradual migration:
+
+```python
+def get_word_quality(word):
+    """Gradual migration approach"""
+    if word in collaborative_scores:
+        # Use collaborative score if available
+        return collaborative_scores[word] / 100.0
+    elif word in subtlex_zipf:
+        # Fallback to SUBTLEX
+        return subtlex_zipf[word] / 7.0
+    else:
+        # Final fallback to WordFreq
+        return word_frequency(word, 'en')
+```
+
+This allows testing new vocabulary sources while maintaining compatibility with existing words not found in curated lists.
+
+## Conclusion (Updated After Hands-On Evaluation)
+
+**Key Finding**: Most "crossword-specific" vocabulary lists contain significant amounts of junk data that require extensive cleanup, defeating their supposed advantage over general-purpose sources.
+
+**Recommended Solution**: Combine high-quality general sources instead:
+1. **COCA free sample** (6K words) for core high-quality vocabulary
+2. **Peter Norvig's 100K list** for clean, broad coverage  
+3. **SUBTLEX** for psycholinguistically validated difficulty grading
+4. **Avoid crossword-specific lists** until they improve their curation
+
+This hybrid approach provides:
+- **Clean vocabulary**: No `10THGENCONSOLE`, `zzzzzzzzzzzzzzz`, or `AAAAUTOCLUB` garbage
+- **Academic validation**: COCA and SUBTLEX are research-proven
+- **Industry credibility**: Norvig's list comes from Google's Director of Research
+- **Reasonable coverage**: 6K-100K words should handle most crossword needs
+- **Better difficulty calibration**: Psycholinguistic frequency data beats arbitrary scores
+
+**Next Steps**:
+1. Start with COCA free sample as proof of concept
+2. Extend with filtered SUBTLEX for broader coverage
+3. Validate against Norvig's clean list
+4. Consider COCA full version if budget allows
+
+The investment in clean, research-backed vocabulary data will dramatically improve puzzle quality without the cleanup nightmare of supposedly "crossword-specific" sources.

hack/analyze_norvig_vocabulary.py

@@ -0,0 +1,400 @@
+#!/usr/bin/env python3
+"""
+Statistical Analysis of Norvig Word Count Files
+
+Analyzes a single Norvig word count file (count_1w.txt or count_1w100k.txt) 
+from norvig.com/ngrams/ to understand vocabulary characteristics for crossword generation.
+
+Usage:
+    python analyze_norvig_vocabulary.py <filename>
+    python analyze_norvig_vocabulary.py --help
+
+Examples:
+    python analyze_norvig_vocabulary.py norvig/count_1w100k.txt
+    python analyze_norvig_vocabulary.py norvig/count_1w.txt
+"""
+
+import os
+import sys
+import argparse
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+from collections import Counter, defaultdict
+import seaborn as sns
+from pathlib import Path
+
+# Set style for better plots
+plt.style.use('seaborn-v0_8')
+sns.set_palette("husl")
+
+def parse_arguments():
+    """Parse command line arguments"""
+    parser = argparse.ArgumentParser(
+        description='Analyze Norvig word count files for crossword generation',
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  python analyze_norvig_vocabulary.py norvig/count_1w100k.txt
+  python analyze_norvig_vocabulary.py norvig/count_1w.txt
+  python analyze_norvig_vocabulary.py --help
+
+File formats supported:
+  - count_1w100k.txt: Top 100,000 most frequent words
+  - count_1w.txt: Full word count dataset (1M+ words)
+
+Output:
+  - Comprehensive statistical analysis
+  - 6-panel visualization saved as norvig_comprehensive_analysis.png
+  - Summary statistics printed to console
+        """
+    )
+    
+    parser.add_argument(
+        'filename',
+        help='Path to Norvig word count file (e.g., norvig/count_1w100k.txt)'
+    )
+    
+    return parser.parse_args()
+
+def load_word_counts(filepath):
+    """Load word count file and return dict of {word: count}"""
+    word_counts = {}
+    total_lines = 0
+    
+    print(f"Loading {filepath}...")
+    
+    try:
+        with open(filepath, 'r', encoding='utf-8') as f:
+            for line in f:
+                total_lines += 1
+                parts = line.strip().split('\t')
+                if len(parts) == 2:
+                    word, count = parts
+                    word_counts[word.upper()] = int(count)
+                elif len(parts) == 1 and line.strip():
+                    # Handle case where count might be missing
+                    word = parts[0]
+                    word_counts[word.upper()] = 1
+        
+        print(f"✅ Loaded {len(word_counts):,} words from {filepath}")
+        return word_counts
+    
+    except FileNotFoundError:
+        print(f"❌ File not found: {filepath}")
+        return {}
+    except Exception as e:
+        print(f"❌ Error loading {filepath}: {e}")
+        return {}
+
+def analyze_word_lengths(words):
+    """Analyze distribution of word lengths"""
+    lengths = [len(word) for word in words]
+    length_dist = Counter(lengths)
+    
+    return lengths, length_dist
+
+def classify_difficulty(rank, total_words):
+    """Classify word difficulty based on frequency rank"""
+    if rank <= total_words * 0.05:  # Top 5%
+        return "Very Easy"
+    elif rank <= total_words * 0.20:  # Top 20%  
+        return "Easy"
+    elif rank <= total_words * 0.60:  # Top 60%
+        return "Medium"
+    elif rank <= total_words * 0.85:  # Top 85%
+        return "Hard"
+    else:
+        return "Very Hard"
+
+def create_comprehensive_analysis(word_counts, filename, base_dir):
+    """Create comprehensive statistical analysis with readable plots"""
+    
+    # Create figure with subplots - 2x3 layout with good spacing
+    fig = plt.figure(figsize=(18, 12))
+    fig.suptitle(f'Norvig Word Count Analysis - {filename}', 
+                 fontsize=16, fontweight='bold', y=0.95)
+    
+    # Convert to sorted lists for analysis
+    words = list(word_counts.keys())
+    counts = list(word_counts.values())
+    ranks = list(range(1, len(counts) + 1))
+    
+    # 1. Zipf's Law Analysis (log-log plot)
+    ax1 = plt.subplot(2, 3, 1)
+    plt.loglog(ranks, counts, 'b-', alpha=0.7, linewidth=2)
+    plt.xlabel('Rank (log scale)')
+    plt.ylabel('Frequency (log scale)')
+    plt.title('Zipf\'s Law Validation', fontweight='bold')
+    plt.grid(True, alpha=0.3)
+    
+    # Add theoretical Zipf line for comparison
+    theoretical_zipf = [counts[0] / r for r in ranks]
+    plt.loglog(ranks, theoretical_zipf, 'r--', alpha=0.5, label='Theoretical')
+    plt.legend()
+    
+    # 2. Word Length Distribution
+    ax2 = plt.subplot(2, 3, 2)
+    lengths, length_dist = analyze_word_lengths(words)
+    lengths_list = sorted(length_dist.keys())
+    counts_list = [length_dist[l] for l in lengths_list]
+    
+    bars = plt.bar(lengths_list, counts_list, alpha=0.7, color='skyblue', edgecolor='navy')
+    plt.xlabel('Word Length (characters)')
+    plt.ylabel('Number of Words')
+    plt.title('Word Length Distribution', fontweight='bold')
+    
+    # Highlight crossword-suitable range (3-12 letters)
+    for i, bar in enumerate(bars):
+        if 3 <= lengths_list[i] <= 12:
+            bar.set_color('lightgreen')
+        elif lengths_list[i] < 3 or lengths_list[i] > 15:
+            bar.set_color('lightcoral')
+    
+    plt.axvspan(3, 12, alpha=0.2, color='green', label='Crossword Range')
+    plt.legend()
+    
+    # 3. Difficulty Distribution
+    ax3 = plt.subplot(2, 3, 3)
+    difficulty_dist = defaultdict(int)
+    for rank in ranks:
+        difficulty = classify_difficulty(rank, len(ranks))
+        difficulty_dist[difficulty] += 1
+    
+    diff_labels = list(difficulty_dist.keys())
+    diff_counts = list(difficulty_dist.values())
+    colors = ['darkgreen', 'green', 'orange', 'red', 'darkred']
+    
+    wedges, texts, autotexts = plt.pie(diff_counts, labels=diff_labels, autopct='%1.1f%%', 
+                                      colors=colors[:len(diff_labels)], startangle=90)
+    plt.title('Difficulty Distribution', fontweight='bold')
+    
+    # 4. Cumulative Frequency Coverage
+    ax4 = plt.subplot(2, 3, 4)
+    cumulative_freq = np.cumsum(counts)
+    total_freq = cumulative_freq[-1]
+    coverage_pct = (cumulative_freq / total_freq) * 100
+    
+    plt.plot(ranks, coverage_pct, 'g-', linewidth=2)
+    plt.xlabel('Vocabulary Size')
+    plt.ylabel('Coverage (%)')
+    plt.title('Cumulative Coverage', fontweight='bold')
+    plt.grid(True, alpha=0.3)
+    
+    # Add key milestone markers
+    milestones = [1000, 5000, 10000, 25000, 50000]
+    for milestone in milestones:
+        if milestone < len(coverage_pct):
+            plt.axvline(x=milestone, color='red', linestyle='--', alpha=0.5)
+    
+    # 5. Crossword Suitability
+    ax5 = plt.subplot(2, 3, 5)
+    crossword_suitable = {word: count for word, count in word_counts.items() 
+                         if 3 <= len(word) <= 12 and word.isalpha()}
+    
+    total_words = len(word_counts)
+    suitable_words = len(crossword_suitable)
+    unsuitable_words = total_words - suitable_words
+    
+    labels = [f'Suitable\n{suitable_words:,}', f'Not Suitable\n{unsuitable_words:,}']
+    sizes = [suitable_words, unsuitable_words]
+    colors = ['lightgreen', 'lightcoral']
+    
+    wedges, texts, autotexts = plt.pie(sizes, labels=labels, autopct='%1.1f%%', colors=colors, startangle=90)
+    plt.title('Crossword Suitability', fontweight='bold')
+    
+    # 6. Difficulty Categories for Crosswords
+    ax6 = plt.subplot(2, 3, 6)
+    
+    # Define crossword difficulty thresholds
+    easy_threshold = 5000
+    medium_threshold = 25000
+    
+    easy_words = sum(1 for i, word in enumerate(words[:easy_threshold]) if 3 <= len(word) <= 12 and i < len(words))
+    medium_words = sum(1 for i, word in enumerate(words[easy_threshold:medium_threshold]) if 3 <= len(word) <= 12 and (i + easy_threshold) < len(words))
+    hard_words = sum(1 for i, word in enumerate(words[medium_threshold:]) if 3 <= len(word) <= 12 and (i + medium_threshold) < len(words))
+    
+    categories = ['Easy', 'Medium', 'Hard']
+    word_counts_cat = [easy_words, medium_words, hard_words]
+    colors_cat = ['lightgreen', 'gold', 'lightcoral']
+    
+    bars = plt.bar(categories, word_counts_cat, color=colors_cat, alpha=0.8)
+    plt.ylabel('Crossword Words')
+    plt.title('Difficulty Categories\n(Based on Frequency Rank)', fontweight='bold')
+    
+    # Add value labels on bars
+    for bar, count in zip(bars, word_counts_cat):
+        height = bar.get_height()
+        if height > 0:
+            plt.text(bar.get_x() + bar.get_width()/2, height + max(word_counts_cat)*0.02,
+                    f'{count:,}', ha='center', va='bottom', fontweight='bold')
+    
+    # Add explanation text box with examples
+    # Get some example words for each category
+    easy_examples = [w for i, w in enumerate(words[:100]) if 3 <= len(w) <= 12][:3]
+    medium_examples = [w for i, w in enumerate(words[7000:12000]) if 3 <= len(w) <= 12][:3]  
+    hard_examples = [w for i, w in enumerate(words[30000:35000]) if 3 <= len(w) <= 12][:3]
+    
+    explanation = (f'Easy: Ranks 1-5,000 (most frequent)\n'
+                   f'  e.g., {", ".join(easy_examples[:3])}\n'
+                   f'Medium: Ranks 5,001-25,000\n'
+                   f'  e.g., {", ".join(medium_examples[:3])}\n'
+                   f'Hard: Ranks 25,001+ (least frequent)\n'
+                   f'  e.g., {", ".join(hard_examples[:3])}\n\n'
+                   'Lower rank = higher frequency = easier')
+    
+    plt.text(0.98, 0.98, explanation, transform=ax6.transAxes, 
+             fontsize=8, verticalalignment='top', horizontalalignment='right',
+             bbox=dict(boxstyle='round,pad=0.5', facecolor='lightblue', alpha=0.9))
+    
+    # Adjust layout with proper spacing
+    plt.subplots_adjust(left=0.08, bottom=0.08, right=0.95, top=0.88, wspace=0.35, hspace=0.45)
+    
+    # Save the comprehensive analysis with filename in the output name
+    # Extract base name and create clean output filename
+    if 'count_1w100k' in filename:
+        output_name = 'norvig_analysis_100k.png'
+    elif 'count_1w.txt' in filename:
+        output_name = 'norvig_analysis_full.png'
+    else:
+        # Fallback for any other filename - make it filesystem safe
+        safe_name = filename.replace('.txt', '').replace('/', '_').replace('count_', '')
+        output_name = f'norvig_analysis_{safe_name}.png'
+    
+    output_path = base_dir / output_name
+    plt.savefig(output_path, dpi=300, bbox_inches='tight')
+    print(f"📊 Comprehensive analysis saved to: {output_path}")
+    
+    return fig, crossword_suitable
+
+def print_summary_statistics(word_counts, filename, crossword_suitable):
+    """Print comprehensive summary statistics"""
+    
+    print("\n" + "="*80)
+    print("📊 NORVIG VOCABULARY STATISTICAL ANALYSIS")
+    print(f"📁 File: {filename}")
+    print("="*80)
+    
+    # Basic statistics
+    total_words = len(word_counts)
+    total_frequency = sum(word_counts.values())
+    
+    print(f"\n📚 BASIC STATISTICS:")
+    print(f"   • Total words: {total_words:,}")
+    print(f"   • Total frequency: {total_frequency:,}")
+    print(f"   • Average frequency: {total_frequency/total_words:.2f}")
+    
+    # Word length analysis
+    lengths, length_dist = analyze_word_lengths(word_counts.keys())
+    avg_length = np.mean(lengths)
+    crossword_length_words = sum(count for length, count in length_dist.items() if 3 <= length <= 12)
+    crossword_length_pct = (crossword_length_words / total_words) * 100
+    
+    print(f"\n📏 WORD LENGTH ANALYSIS:")
+    print(f"   • Average word length: {avg_length:.1f} characters")
+    print(f"   • Words 3-12 characters: {crossword_length_words:,} ({crossword_length_pct:.1f}%)")
+    print(f"   • Most common lengths: {sorted(length_dist.items(), key=lambda x: x[1], reverse=True)[:5]}")
+    
+    # Crossword suitability
+    suitable_count = len(crossword_suitable)
+    suitable_pct = (suitable_count / total_words) * 100
+    suitable_freq = sum(crossword_suitable.values())
+    suitable_freq_pct = (suitable_freq / total_frequency) * 100
+    
+    print(f"\n🧩 CROSSWORD SUITABILITY:")
+    print(f"   • Suitable words (3-12 letters, alphabetic): {suitable_count:,} ({suitable_pct:.1f}%)")
+    print(f"   • Suitable word frequency coverage: {suitable_freq_pct:.1f}%")
+    
+    # Difficulty distribution for crosswords
+    easy_words = len([w for w, c in list(crossword_suitable.items())[:5000]])
+    medium_words = len([w for w, c in list(crossword_suitable.items())[5000:25000]])
+    hard_words = len([w for w, c in list(crossword_suitable.items())[25000:]])
+    
+    print(f"\n🎯 CROSSWORD DIFFICULTY DISTRIBUTION:")
+    print(f"   • Easy (rank 1-5K): {easy_words:,} words")
+    print(f"   • Medium (rank 5K-25K): {medium_words:,} words") 
+    print(f"   • Hard (rank 25K+): {hard_words:,} words")
+    
+    # Top and bottom words examples
+    words_list = list(word_counts.keys())
+    print(f"\n🔝 TOP 10 MOST FREQUENT WORDS:")
+    for i, word in enumerate(words_list[:10], 1):
+        print(f"   {i:2d}. {word:<12} ({word_counts[word]:,})")
+    
+    print(f"\n🔚 BOTTOM 10 LEAST FREQUENT WORDS:")
+    for i, word in enumerate(words_list[-10:], 1):
+        print(f"   {i:2d}. {word:<12} ({word_counts[word]:,})")
+    
+    # Zipf's law validation
+    words_list = list(word_counts.keys())
+    counts_list = list(word_counts.values())
+    
+    # Calculate correlation coefficient for log-log relationship
+    log_ranks = np.log(range(1, len(counts_list) + 1))
+    log_freqs = np.log(counts_list)
+    correlation = np.corrcoef(log_ranks, log_freqs)[0, 1]
+    
+    print(f"\n📈 ZIPF'S LAW VALIDATION:")
+    print(f"   • Log-log correlation: {correlation:.4f}")
+    print(f"   • Zipf compliance: {'✅ Excellent' if abs(correlation) > 0.95 else '⚠️ Moderate' if abs(correlation) > 0.8 else '❌ Poor'}")
+    
+    # Recommendations
+    print(f"\n💡 RECOMMENDATIONS FOR CROSSWORD GENERATION:")
+    print(f"   • Dataset size: {total_words:,} words with excellent coverage")
+    print(f"   • Filter to 3-12 letters: Reduces to {suitable_count:,} words ({suitable_pct:.1f}%)")
+    print(f"   • Difficulty thresholds (for crossword-suitable words):")
+    print(f"     - Easy: ranks 1-5,000 ({easy_words:,} suitable words)")
+    print(f"     - Medium: ranks 5,001-25,000 ({medium_words:,} suitable words)")
+    print(f"     - Hard: ranks 25,001+ ({hard_words:,} suitable words)")
+    print(f"   • Quality: ✅ No garbage entries (unlike crossword-specific lists)")
+    print(f"   • Source credibility: ✅ Peter Norvig (Google) + Google Books corpus")
+    
+    print("="*80)
+
+def main():
+    """Main analysis function"""
+    
+    # Parse command line arguments
+    args = parse_arguments()
+    
+    # File paths
+    base_dir = Path(__file__).parent
+    input_file = Path(args.filename)
+    
+    # Make path relative to script directory if not absolute
+    if not input_file.is_absolute():
+        input_file = base_dir / input_file
+    
+    print("🔍 Norvig Vocabulary Statistical Analysis")
+    print("=" * 50)
+    print(f"📁 Analyzing: {input_file}")
+    
+    # Load data
+    word_counts = load_word_counts(input_file)
+    
+    if not word_counts:
+        print(f"❌ Could not load word list from {input_file}. Please check file path.")
+        return
+    
+    # Create comprehensive analysis
+    fig, crossword_suitable = create_comprehensive_analysis(word_counts, input_file.name, base_dir)
+    
+    # Print summary statistics
+    print_summary_statistics(word_counts, input_file.name, crossword_suitable)
+    
+    # Don't show plot interactively in CLI, just save it
+    # plt.show()  # Comment out for CLI usage
+    
+    # Generate the same output filename logic for final message
+    if 'count_1w100k' in input_file.name:
+        output_name = 'norvig_analysis_100k.png'
+    elif 'count_1w.txt' in input_file.name:
+        output_name = 'norvig_analysis_full.png'
+    else:
+        safe_name = input_file.name.replace('.txt', '').replace('/', '_').replace('count_', '')
+        output_name = f'norvig_analysis_{safe_name}.png'
+    
+    print(f"\n✅ Analysis complete! Check {base_dir}/{output_name} for detailed plots.")
+
+if __name__ == "__main__":
+    main()