CELERY_SCHEDULING_SETUP.md

# Celery Scheduling Setup Guide

This guide explains how to set up and use the Celery scheduling system with your Lin application.

## Overview

The updated `start_app.py` now automatically starts both the Flask application and Celery components (worker and beat scheduler) when you run the application. This ensures that your scheduled tasks will execute properly.

## Prerequisites

### 1. Redis Server
Celery requires Redis as a message broker. Make sure Redis is installed and running:

**Windows:**
```bash
# Install Redis (if not installed)
# Download from https://github.com/microsoftarchive/redis/releases

# Start Redis server
redis-server
```

**Linux/Mac:**
```bash
# Install Redis
sudo apt-get install redis-server  # Ubuntu/Debian
brew install redis                 # macOS

# Start Redis
sudo systemctl start redis
sudo systemctl enable redis
```

### 2. Hugging Face Spaces / Production Deployment
For Hugging Face Spaces or production deployments where you can't run Redis directly:

**Option A: Use Redis Cloud (Recommended)**
1. Create a free Redis Cloud account at https://redislabs.com/try-free/
2. Create a Redis database (free tier available)
3. Update your `.env` file:
```env
CELERY_BROKER_URL="redis://your-redis-host:port/0"
CELERY_RESULT_BACKEND="redis://your-redis-host:port/0"
```

**Option B: Use Docker Compose**
```yaml
# docker-compose.yml
version: '3.8'
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes

volumes:
  redis_data:
```

**Option C: Skip Celery (Basic Mode)**
If Redis is not available, the Flask app will start without Celery functionality. Schedules will be saved but won't execute automatically.

### 2. Python Dependencies
Install the required packages:
```bash
pip install -r backend/requirements.txt
```

## Starting the Application

### Using start_app.py (Recommended)
```bash
python start_app.py
```

This will:
1. Check Redis connection
2. Start Celery worker in the background
3. Start Celery beat scheduler in the background
4. Start the Flask application

### Using Backend Scripts (Alternative)
```bash
# Start both worker and beat
cd backend
python start_celery.py all

# Or start individually
python start_celery.py worker  # Start Celery worker
python start_celery.py beat   # Start Celery beat scheduler
```

## Configuration

### Environment Variables
Make sure these are set in your `.env` file:

```env
# Supabase configuration
SUPABASE_URL="your_supabase_url"
SUPABASE_KEY="your_supabase_key"

# Redis configuration (if not using defaults)
CELERY_BROKER_URL="redis://localhost:6379/0"
CELERY_RESULT_BACKEND="redis://localhost:6379/0"

# Scheduler configuration
SCHEDULER_ENABLED=True
```

### Celery Configuration
The unified configuration is in `backend/celery_config.py`:

```python
celery_app.conf.beat_schedule = {
    'load-schedules': {
        'task': 'backend.celery_tasks.schedule_loader.load_schedules_task',
        'schedule': crontab(minute='*/5'),  # Every 5 minutes
    },
}
```

## How Scheduling Works

### 1. Schedule Loading
- **Immediate Updates**: When you create or delete a schedule via the API, Celery Beat is updated immediately
- **Periodic Updates**: Celery Beat also runs every 5 minutes as a backup
- Executes `load_schedules_task`
- Fetches schedules from Supabase database
- Creates individual periodic tasks for each schedule

### 2. Task Execution
- **Content Generation**: Runs 5 minutes before scheduled time
- **Post Publishing**: Runs at the scheduled time
- Tasks are queued in appropriate queues (content, publish)

### 3. Database Integration
- Uses Supabase for schedule storage
- Automatically creates tasks based on schedule data
- Handles social network authentication

## Monitoring and Debugging

### Checking Celery Status
```bash
# Check worker status
celery -A celery_config inspect stats

# Check scheduled tasks
celery -A celery_config inspect scheduled

# Check active tasks
celery -A celery_config inspect active
```

### Viewing Logs
- **Flask Application**: Check console output
- **Celery Worker**: Look for worker process logs
- **Celery Beat**: Look for beat process logs

### Common Issues

**1. Redis Connection Failed**
```
Error: Error 111 connecting to localhost:6379. Connection refused

Solutions:
1. Start Redis server locally (development):
   Windows: redis-server
   Linux/Mac: sudo systemctl start redis

2. Use Redis Cloud (production/Hugging Face):
   - Create free Redis Cloud account
   - Update .env with Redis Cloud URL
   - Set CELERY_BROKER_URL and CELERY_RESULT_BACKEND

3. Use Docker Compose:
   version: '3.8'
   services:
     redis:
       image: redis:7-alpine
       ports: ["6379:6379"]
       volumes: [redis_data:/data]
   volumes: redis_data:

4. Skip Celery (basic mode):
   - App will start without scheduling functionality
   - Schedules saved but won't execute automatically
```

**2. Tasks Not Executing**
```bash
# Check if Celery worker is running
celery -A celery_config inspect ping

# Check if beat scheduler is running
ps aux | grep celery
```

**3. Schedule Not Loading**
- Check Supabase database connection
- Verify schedule data in database
- Check task registration in Celery

## Testing the Scheduling System

### Manual Testing
```python
# Test schedule loading task
from backend.celery_tasks.schedule_loader import load_schedules_task
result = load_schedules_task()
print(result)
```

### API Testing (Recommended)
1. **Create a schedule via the API**:
   ```bash
   curl -X POST http://localhost:5000/api/schedules/ \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer YOUR_JWT_TOKEN" \
     -d '{
       "social_network": "1",
       "schedule_time": "09:00",
       "days": ["Monday", "Wednesday", "Friday"]
     }'
   ```
   
2. **Check the response**: You should see a `celery_update_task_id` field indicating the scheduler was updated immediately

3. **Verify in Celery**: Check if the individual tasks were created:
   ```bash
   celery -A celery_config inspect scheduled
   ```

### Database Testing
1. Add a schedule directly in the Supabase database
2. Wait 5 minutes for the loader task to run (or trigger via API)
3. Check if individual tasks were created
4. Verify task execution times

## Production Deployment

### Using Docker (Recommended for Hugging Face Spaces)
```bash
# Build the Docker image
docker build -t lin-app .

# Run the container
docker run -p 7860:7860 lin-app

# For Hugging Face Spaces deployment:
# 1. Update your Dockerfile (already done above)
# 2. Push to Hugging Face Spaces
# 3. The container will automatically start Redis + your app
```

### Using Docker Compose (Local Development)
```yaml
# docker-compose.yml
version: '3.8'
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    volumes: [redis_data:/data]
    command: redis-server --appendonly yes
  
  app:
    build: .
    ports: ["7860:7860"]
    depends_on: [redis]
    environment:
      - REDIS_URL=redis://redis:6379/0
    volumes: [./:/app]

volumes: redis_data:
```

```bash
# Start all services
docker-compose up -d

# Check logs
docker-compose logs -f
```

### Using Docker
```bash
# Build and start all services
docker-compose up -d

# Check logs
docker-compose logs -f
```

### Using Supervisor (Linux)
Create `/etc/supervisor/conf.d/lin.conf`:
```ini
[program:lin_worker]
command=python start_app.py
directory=/path/to/lin
autostart=true
autorestart=true
user=www-data
environment=PATH="/path/to/venv/bin"

[program:lin_beat]
command=python -m celery -A celery_config beat
directory=/path/to/lin
autostart=true
autorestart=true
user=www-data
environment=PATH="/path/to/venv/bin"
```

## Troubleshooting Checklist

1. ✅ Redis server is running
2. ✅ All Python dependencies are installed
3. ✅ Environment variables are set correctly
4. ✅ Supabase database connection works
5. ✅ Celery worker is running
6. ✅ Celery beat scheduler is running
7. ✅ Schedule data exists in database
8. ✅ Tasks are properly registered
9. ✅ Task execution permissions are correct

## Support

If you encounter issues:
1. Check this guide first
2. Review the logs for error messages
3. Verify all prerequisites are met
4. Test components individually
5. Check the Celery documentation

For additional help, refer to the Celery documentation at: https://docs.celeryq.dev/