Initial commit with essential files

.streamlit/config.toml

@@ -0,0 +1,4 @@
+[theme]
+base="dark"
+primaryColor="#865bf1"
+font="monospace"

README.md

@@ -0,0 +1,246 @@
+# AI Lip Sync
+
+![Screenshot 2024-01-22 at 03-03-09 app · Streamlit](https://github.com/Aml-Hassan-Abd-El-hamid/AI-Lip-Sync/assets/66205928/d35f379f-f1ca-46e5-a113-bfa3f3c4c2f9)
+
+An AI-powered application that synchronizes lip movements with audio input, built with Wav2Lip and Streamlit.
+
+## Features
+
+- **Multiple Avatar Options**: Choose from built-in avatars or upload your own image/video
+- **Audio Input Flexibility**: Record audio directly or upload WAV/MP3 files
+- **Quality Assessment**: Automatic analysis of video and audio quality with recommendations
+- **GPU Acceleration**: Optimized for Apple Silicon (M1/M2) GPUs
+- **Two Animation Modes**: Fast (lips only) or Slow (full face animation)
+- **Video Trimming**: Trim the output video to remove unwanted portions
+
+## Quick Setup Guide
+
+### Prerequisites
+
+- Python 3.9+ 
+- ffmpeg (for audio processing)
+- Git LFS (optional, for handling large model files)
+
+### Installation
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/yourusername/ai-lip-sync-app.git
+   cd ai-lip-sync-app
+   ```
+
+2. Create and activate a virtual environment:
+   ```bash
+   python -m venv .venv
+   # On macOS/Linux
+   source .venv/bin/activate
+   # On Windows
+   .venv\Scripts\activate
+   ```
+
+3. Install Python dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. Install system dependencies:
+   ```bash
+   # On Ubuntu/Debian
+   sudo apt-get update
+   sudo apt-get install $(cat packages.txt)
+   
+   # On macOS with Homebrew
+   brew install ffmpeg
+   ```
+
+5. Run the application:
+   ```bash
+   python -m streamlit run app.py
+   ```
+   
+   > **Note**: If you encounter a "streamlit: command not found" error, always use `python -m streamlit run app.py` instead of `streamlit run app.py`
+
+The application will automatically download the required model files on first run.
+
+## Usage Guide
+
+1. **Choose Avatar Source**:
+   - Select from built-in avatars or upload your own image/video
+   - For best results, use clear frontal face images/videos
+
+2. **Provide Audio**:
+   - Record directly using your microphone
+   - Upload WAV or MP3 files
+
+3. **Quality Assessment**:
+   - The app will automatically analyze your uploaded video and audio
+   - Review the quality analysis and recommendations
+   - Make adjustments if needed for better results
+
+4. **Generate Animation**:
+   - Choose "Fast animate" for quicker processing (lips only)
+   - Choose "Slower animate" for more realistic results (full face)
+
+5. **View and Edit Results**:
+   - The generated video will appear in the app
+   - Use the trim feature to remove unwanted portions from the start or end
+   - Download the original or trimmed version to your computer
+
+## Video Trimming Feature
+
+The app now includes a video trimming capability:
+
+- After generating a lip-sync video, you'll see trimming options below the result
+- Use the sliders to select the start and end times for your trimmed video
+- Click "Trim Video" to create a shortened version
+- Both original and trimmed videos can be downloaded directly from the app
+
+## Quality Assessment Feature
+
+The app now includes automatic quality assessment for uploaded videos and audio:
+
+### Video Analysis:
+- Resolution check (higher resolution = better results)
+- Face detection (confirms a face is present and properly sized)
+- Frame rate analysis
+- Overall quality score with specific recommendations
+
+### Audio Analysis:
+- Speech detection (confirms speech is present)
+- Volume level assessment
+- Silence detection
+- Overall quality score with specific recommendations
+
+## Troubleshooting
+
+- **"No face detected" error**: Ensure your video has a clear, well-lit frontal face
+- **Poor lip sync results**: Try using higher quality audio with clear speech
+- **Performance issues**: For large videos, try the "Fast animate" option or use a smaller video clip
+- **Memory errors**: Close other applications to free up memory, or use a machine with more RAM
+
+## Technical Details
+
+The project is built on the Wav2Lip model with several optimizations:
+- Apple Silicon (M1/M2) GPU acceleration using MPS backend
+- Automatic video resolution scaling for large videos
+- Memory optimizations for processing longer videos
+- Quality assessment using OpenCV and librosa
+
+## Original Project Background
+
+The project started as a part of an interview process with some company, I received an email with the following task:
+
+Assignment Object:<br>
+          &emsp;&emsp;&emsp;&emsp;Your task is to develop a lip-syncing model using machine learning
+          techniques. It takes an input image and audio and then generates a video
+          where the image appears to lip sync with the provided audio. You have to
+          develop this task using python3.
+
+Requirements:<br>
+        &emsp;&emsp;&emsp;&emsp;● Avatar / Image : Get one AI-generated avatar, the avatar may be for a<br>
+        &emsp;&emsp;&emsp;&emsp;man, woman, old man, old lady or a child. Ensure that the avatar is<br>
+        &emsp;&emsp;&emsp;&emsp;created by artificial intelligence and does not represent real<br>
+        &emsp;&emsp;&emsp;&emsp;individuals.<br>
+        &emsp;&emsp;&emsp;&emsp;● Audio : Provide two distinct and clear audio recordings—one in Arabic<br>
+        &emsp;&emsp;&emsp;&emsp;and the other in English. The duration of each audio clip should be<br>
+        &emsp;&emsp;&emsp;&emsp;no less than 30 seconds and no more than 1 minute.<br>
+        &emsp;&emsp;&emsp;&emsp;● Lip-sync model: Develop a lip-syncing model to synchronise the lip<br>
+        &emsp;&emsp;&emsp;&emsp;movements of the chosen avatar with the provided audio. Ensure the<br>
+        &emsp;&emsp;&emsp;&emsp;model demonstrates proficiency in accurately aligning lip motions<br>
+        &emsp;&emsp;&emsp;&emsp;with the spoken words in both Arabic and English.<br>
+        &emsp;&emsp;&emsp;&emsp;Hint : You can refer to state of the art models in lip-syncing.<br>
+         
+I was given about 96 hours to accomplish this task, I spent the first 12 hours sick with a very bad flu and no proper internet connection so I had 84 hours!<br>
+After submitting the task on time, I took more time to deploy the project on Streamlight, as I thought it was a fun project and would be a nice addition to my CV:)  
+
+Given the provided hint from the company, "You can refer to state-of-the-art models in lip-syncing.", I started looking into the available open-source pre-trained model that can accomplish this task and most available resources pointed towards **Wav2Lip**. I found a couple of interesting tutorials for that model that I will share below.
+
+### How to run the application locally:<br>
+
+1- clone the repo to your local machine.<br>
+2- open your terminal inside the project folder and run the following command: `pip install -r requirements.txt` and then run this command `sudo xargs -a packages.txt apt-get install` to install the needed modules and packages.<br>
+3- open your terminal inside the project folder and run the following command: `streamlit run app.py` to run the streamlit application.<br>
+
+### Things I changed in the wav2lip and why:<br>
+
+In order to work with and deploy the wav2lip model I had to make the following changes:<br>
+1- Changed the `_build_mel_basis()` function in `audio.py`, I had to do that to be able to work with `librosa>=0.10.0` package, check this [issue](https://github.com/Rudrabha/Wav2Lip/issues/550) for more details.<br>
+2- Changed the `main()` function at the `inferance.py` to directly take an output from the `app.py` instead of using the command line arguments.<br>
+3- I took the `load_model(path)` function and added it to `app.py` and added `@st.cache_data` in order to only load the model once, instead of using it multiple times, I also modified it<br>
+4- Deleted the unnecessary files like the checkpoints to make the Streamlit website deployment easier.<br>
+5- Since I'm using Streamlit for deployment and Streamlit Cloud doesn't support GPU, I had to change the device to work with `cpu` instead of `cuda`.<br>
+6- I made other minor changes like changing the path to a file or modifying import statements.
+
+### Issues I had with Streamlit, during the deployment:
+
+This part is a documentation for me, just in case, I need to face an issue in the future and also could be helpful for any poor soul who would have to work with Streamlit:
+
+1- 
+```
+Error downloading object: wav2lip/checkpoints/wav2lip_gan.pth (ca9ab7b): Smudge error: Error downloading wav2lip/checkpoints/wav2lip_gan.pth (ca9ab7b7b812c0e80a6e70a5977c545a1e8a365a6c49d5e533023c034d7ac3d8): batch request: [email protected]: Permission denied (publickey).: exit status 255
+
+Errors logged to /mount/src/ai-lip-sync/.git/lfs/logs/20240121T212252.496674
+```
+This essentially Streamlit telling you that it can't handle that big file, upload it to Google Drive, and then load it using Python code later, and no `git lfs` won't solve the problem :)<br> 
+A ground rule that I learned here is: that the lighter you make your app, the better and faster it is to deploy it.<br>
+I opened a topic with that issue on the Streamlit forum, right [here](https://discuss.streamlit.io/t/file-upload-fails-with-error-downloading-object-wav2lip-checkpoints-wav2lip-gan-pth-ca9ab7b/60261)<br>
+
+2- Other issues that I faced a lot were dependency issues -lots of them- and that was mostly due to the fact that I depended on `pipreqs` to write down my `requirements.txt`, that `pipreqs` missed up my modules, it added unneeded ones and missed others, unfortunately, it took me some time to discover that and really slowed me down.
+
+3- 
+```
+ ImportError: libGL.so.1: cannot open shared object file: No such file or directory
+```
+I faced that problem during importing `cv2` -`openCv`- and the solution was to install `libgl1-mesa-dev` and some other packages using `apt`, you can't just add such packages to the `requirements.txt`, you need to create a file named `packages.txt` to do so.
+
+4- Streamlit can't handle heavy processing, I discovered that when I tried to deploy the `slow animation` button to process video input alongside recording to get more accurate lip-syncing, the application failed directly when I used that button -and I tried to use it twice :)-, and that kinda make sense as Streamlit doesn't have a GPU or even a high ram space -I don't have a good GPU but I have about 64GB ram which was enough to run that function locally- and to solve that issue, I initiated another branch to contain the deployment version that doesn't have the `slow animation` button and used that branch for deployment while kept the main branch containing that button.
+
+**Pushing the checkpoints files:**<br>
+
+Given the size of those kind of files, There are 2 ways to handle that. 
+
+At the start, I had to use git lfs, here's how to do it:<br>
+
+1- Follow the installation instructions that are suitable for your system from [here](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) <br>
+2- Use the command `git lfs track "*.pth"` to let git lfs know that those are your big files.<br>
+3- When pushing from the command line -I usually use VS code but it usually doesn't work with big files like `.pth` files- you need to generate a personal access token, to do so, follow the instructions from [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token), and then copy the token<br>
+4- When pushing the file from the terminal you will be asked to pass a password, don't pass your GitHub profile password, instead pass your personal access token that you got from step 3.
+
+But then Streamlit wasn't capable of even pulling the repo! so I uploaded the model checkpoints and some other files to Google Drive, put them in a public folder, and then used a module called gdown to download those folders when needed! here's a [link](https://github.com/wkentaro/gdown) to that gdown, it's straightforward to use and install.
+
+
+**Video preview of the application:**<br>
+
+**fast animation version**<br>
+Notice how only the lips are moving.
+
+English version:
+
+https://github.com/Aml-Hassan-Abd-El-hamid/AI-Lip-Sync/assets/66205928/36577ccb-5ec6-4bb4-b7ff-44bb52a4f984
+
+Arabic version:
+
+
+
+https://github.com/Aml-Hassan-Abd-El-hamid/ai-lip-sync-app/assets/66205928/4346aa6d-ea4e-400e-9124-1cce06b049df
+
+
+
+**slower animation version**<br>
+Notice how the eye and the whole face are moving instead of only the lips.<br>
+
+Unfortunately, Streamlit can't handle the computational power that the slower animation version requires and that's why I made it only available on the offline version, which means that you need to run the application locally to try that version.
+
+English version:
+
+https://github.com/Aml-Hassan-Abd-El-hamid/AI-Lip-Sync/assets/66205928/26740856-52e5-4fe7-868d-3b9341e97064
+
+Arabic version:
+
+
+
+https://github.com/Aml-Hassan-Abd-El-hamid/ai-lip-sync-app/assets/66205928/ba97daca-b30d-4179-9387-a382abbca3ba
+
+
+
+The only difference between the fast and slow versions of animation here is the fact that the fast version passes only a photo while the slow version passes a video instead.

app.py

@@ -0,0 +1,1000 @@
+import os
+import streamlit as st
+from streamlit_image_select import image_select
+import torch
+from streamlit_mic_recorder import mic_recorder
+from wav2lip import inference 
+from wav2lip.models import Wav2Lip
+import gdown
+import warnings
+import cv2
+import numpy as np
+import librosa
+from pathlib import Path
+import subprocess
+import time
+from PIL import Image
+import matplotlib.pyplot as plt
+import sys
+import threading
+import concurrent.futures
+
+# Suppress warnings
+warnings.filterwarnings('ignore')
+
+# More comprehensive fix for Streamlit file watcher issues with PyTorch
+os.environ['STREAMLIT_WATCH_IGNORE'] = 'torch'
+if 'torch' in sys.modules:
+    sys.modules['torch'].__path__ = type('', (), {'_path': []})()
+
+# Check if MPS (Apple Silicon GPU) is available, otherwise use CPU
+if torch.backends.mps.is_available():
+    device = 'mps'
+    # Enable memory optimization for Apple Silicon
+    torch.mps.empty_cache()
+    # Set the memory format to optimize for M2 Max
+    torch._C._set_cudnn_benchmark(True)
+    st.success("Using Apple M2 Max GPU for acceleration with optimized settings!")
+else:
+    device = 'cpu'
+    st.warning("Using CPU for inference (slower). GPU acceleration not available.")
+
+print(f"Using {device} for inference.")
+
+# Add functions to analyze video and audio quality
+def analyze_video_quality(file_path):
+    """Analyze video quality and detect faces for better user guidance"""
+    try:
+        # Open the video file
+        video = cv2.VideoCapture(file_path)
+        
+        # Get video properties
+        width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
+        height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
+        fps = video.get(cv2.CAP_PROP_FPS)
+        frame_count = int(video.get(cv2.CAP_PROP_FRAME_COUNT))
+        duration = frame_count / fps if fps > 0 else 0
+        
+        # Read a frame for face detection
+        success, frame = video.read()
+        if not success:
+            return {
+                "resolution": f"{width}x{height}",
+                "fps": fps,
+                "duration": f"{duration:.1f} seconds",
+                "quality": "Unknown",
+                "face_detected": False,
+                "message": "Could not analyze video content."
+            }
+        
+        # Detect faces using OpenCV's face detector
+        face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
+        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+        faces = face_cascade.detectMultiScale(gray, 1.1, 4)
+        
+        # Determine quality score based on resolution and face detection
+        quality_score = 0
+        
+        # Resolution assessment
+        if width >= 1920 or height >= 1080:  # 1080p or higher
+            resolution_quality = "Excellent"
+            quality_score += 3
+        elif width >= 1280 or height >= 720:  # 720p
+            resolution_quality = "Good"
+            quality_score += 2
+        elif width >= 640 or height >= 480:  # 480p
+            resolution_quality = "Fair"
+            quality_score += 1
+        else:
+            resolution_quality = "Low"
+            
+        # Overall quality assessment
+        face_detected = len(faces) > 0
+        
+        if face_detected:
+            quality_score += 2
+            face_message = "Face detected! ✅"
+            
+            # Check face size relative to frame
+            for (x, y, w, h) in faces:
+                face_area_ratio = (w * h) / (width * height)
+                if face_area_ratio > 0.1:  # Face takes up at least 10% of frame
+                    quality_score += 1
+                    face_size = "Good face size"
+                else:
+                    face_size = "Face may be too small"
+        else:
+            face_message = "No face detected! ⚠️ Lip sync results may be poor."
+            face_size = "N/A"
+        
+        # Determine overall quality
+        if quality_score >= 5:
+            quality = "Excellent"
+        elif quality_score >= 3:
+            quality = "Good"
+        elif quality_score >= 1:
+            quality = "Fair"
+        else:
+            quality = "Poor"
+            
+        # Release video resource
+        video.release()
+        
+        return {
+            "resolution": f"{width}x{height}",
+            "fps": f"{fps:.1f}",
+            "duration": f"{duration:.1f} seconds",
+            "quality": quality,
+            "resolution_quality": resolution_quality,
+            "face_detected": face_detected,
+            "face_message": face_message,
+            "face_size": face_size,
+            "message": get_video_recommendation(quality, face_detected, width, height)
+        }
+        
+    except Exception as e:
+        return {
+            "quality": "Error",
+            "message": f"Could not analyze video: {str(e)}"
+        }
+
+def analyze_audio_quality(file_path):
+    """Analyze audio quality for better user guidance"""
+    try:
+        # Load audio file using librosa
+        y, sr = librosa.load(file_path, sr=None)
+        
+        # Get duration
+        duration = librosa.get_duration(y=y, sr=sr)
+        
+        # Calculate audio features
+        rms = librosa.feature.rms(y=y)[0]
+        mean_volume = np.mean(rms)
+        
+        # Simple speech detection (using energy levels)
+        has_speech = np.max(rms) > 0.05
+        
+        # Check for silence periods
+        silence_threshold = 0.01
+        silence_percentage = np.mean(rms < silence_threshold) * 100
+        
+        # Calculate quality score
+        quality_score = 0
+        
+        # Volume assessment
+        if 0.05 <= mean_volume <= 0.2:
+            volume_quality = "Good volume levels"
+            quality_score += 2
+        elif mean_volume > 0.2:
+            volume_quality = "Audio might be too loud"
+            quality_score += 1
+        else:
+            volume_quality = "Audio might be too quiet"
+            
+        # Speech detection
+        if has_speech:
+            speech_quality = "Speech detected ✅"
+            quality_score += 2
+        else:
+            speech_quality = "Speech may not be clear ⚠️"
+        
+        # Silence assessment (some silence is normal)
+        if silence_percentage < 40:
+            silence_quality = "Good speech-to-silence ratio"
+            quality_score += 1
+        else:
+            silence_quality = "Too much silence detected"
+        
+        # Determine overall quality
+        if quality_score >= 4:
+            quality = "Excellent"
+        elif quality_score >= 2:
+            quality = "Good"
+        elif quality_score >= 1:
+            quality = "Fair"
+        else:
+            quality = "Poor"
+            
+        return {
+            "duration": f"{duration:.1f} seconds",
+            "quality": quality,
+            "volume_quality": volume_quality,
+            "speech_quality": speech_quality,
+            "silence_quality": silence_quality,
+            "message": get_audio_recommendation(quality, has_speech, mean_volume, silence_percentage)
+        }
+        
+    except Exception as e:
+        return {
+            "quality": "Error",
+            "message": f"Could not analyze audio: {str(e)}"
+        }
+
+def get_video_recommendation(quality, face_detected, width, height):
+    """Get recommendations based on video quality"""
+    if not face_detected:
+        return "⚠️ No face detected. For best results, use a video with a clear, well-lit face looking toward the camera."
+    
+    if quality == "Poor":
+        return "⚠️ Low quality video. Consider using a higher resolution video with better lighting and a clearly visible face."
+    
+    if width < 640 or height < 480:
+        return "⚠️ Video resolution is low. For better results, use a video with at least 480p resolution."
+    
+    if quality == "Excellent":
+        return "✅ Great video quality! This should work well for lip syncing."
+    
+    return "✅ Video quality is acceptable for lip syncing."
+
+def get_audio_recommendation(quality, has_speech, volume, silence_percentage):
+    """Get recommendations based on audio quality"""
+    if not has_speech:
+        return "⚠️ Speech may not be clearly detected. For best results, use audio with clear speech."
+    
+    if quality == "Poor":
+        return "⚠️ Low quality audio. Consider using clearer audio with consistent volume levels."
+    
+    if volume < 0.01:
+        return "⚠️ Audio volume is very low. This may result in poor lip sync."
+    
+    if volume > 0.3:
+        return "⚠️ Audio volume is very high. This may cause distortion in lip sync."
+    
+    if silence_percentage > 50:
+        return "⚠️ Audio contains a lot of silence. Lip sync will only work during speech sections."
+    
+    if quality == "Excellent":
+        return "✅ Great audio quality! This should work well for lip syncing."
+    
+    return "✅ Audio quality is acceptable for lip syncing."
+
+#@st.cache_data is used to only load the model once
+#@st.cache_data 
+@st.cache_resource
+def load_model(path):
+    st.write("Please wait for the model to be loaded or it will cause an error")
+    wav2lip_checkpoints_url = "https://drive.google.com/drive/folders/1Sy5SHRmI3zgg2RJaOttNsN3iJS9VVkbg?usp=sharing"
+    if not os.path.exists(path):
+        gdown.download_folder(wav2lip_checkpoints_url, quiet=True, use_cookies=False)
+    st.write("Please wait")
+    model = Wav2Lip()
+    print("Load checkpoint from: {}".format(path))
+    
+    # Optimize model loading for M2 Max
+    if device == 'mps':
+        # Clear cache before loading model
+        torch.mps.empty_cache()
+        
+    # Load model with device mapping
+    checkpoint = torch.load(path, map_location=torch.device(device))
+    s = checkpoint["state_dict"]
+    new_s = {}
+    for k, v in s.items():
+        new_s[k.replace('module.', '')] = v
+    model.load_state_dict(new_s)
+    model = model.to(device)
+    
+    # Set model to evaluation mode and optimize for inference
+    model.eval()
+    if device == 'mps':
+        # Attempt to optimize the model for inference
+        try:
+            # Use torch's inference mode for optimized inference
+            torch._C._jit_set_profiling_executor(False)
+            torch._C._jit_set_profiling_mode(False)
+            print("Applied M2 Max optimizations")
+        except:
+            print("Could not apply all M2 Max optimizations")
+            
+    st.write(f"Model loaded successfully on {device} with optimized settings for M2 Max!")
+    return model
+@st.cache_resource
+def load_avatar_videos_for_slow_animation(path):
+    if not os.path.exists(path):
+        try:
+            os.makedirs(path, exist_ok=True)
+            print(f"Created directory: {path}")
+            
+            avatar_videos_url = "https://drive.google.com/drive/folders/1h9pkU5wenrS2vmKqXBfFmrg-1hYw5s4q?usp=sharing"
+            print(f"Downloading avatar videos from: {avatar_videos_url}")
+            gdown.download_folder(avatar_videos_url, quiet=False, use_cookies=False)
+            print(f"Avatar videos downloaded successfully to: {path}")
+        except Exception as e:
+            print(f"Error downloading avatar videos: {str(e)}")
+            # Create default empty videos if download fails
+            for avatar_file in ["avatar1.mp4", "avatar2.mp4", "avatar3.mp4"]:
+                video_path = os.path.join(path, avatar_file)
+                if not os.path.exists(video_path):
+                    print(f"Creating empty video file: {video_path}")
+                    # Get the matching image
+                    img_key = f"avatars_images/{os.path.splitext(avatar_file)[0]}" + (".jpg" if avatar_file != "avatar3.mp4" else ".png")
+                    try:
+                        # Create a video from the image
+                        img = cv2.imread(img_key)
+                        if img is not None:
+                            # Create a short 5-second video from the image
+                            print(f"Creating video from image: {img_key}")
+                            height, width = img.shape[:2]
+                            output_video = cv2.VideoWriter(video_path, cv2.VideoWriter_fourcc(*'mp4v'), 30, (width, height))
+                            for _ in range(150):  # 5 seconds at 30 fps
+                                output_video.write(img)
+                            output_video.release()
+                        else:
+                            print(f"Could not read image: {img_key}")
+                    except Exception as e:
+                        print(f"Error creating video from image: {str(e)}")
+    else:
+        print(f"Avatar videos directory already exists: {path}")
+        # Check if files exist in the directory
+        files = os.listdir(path)
+        if not files:
+            print(f"No files found in {path}, directory exists but is empty")
+        else:
+            print(f"Found {len(files)} files in {path}: {', '.join(files)}")
+
+
+
+image_video_map = {
+      				"avatars_images/avatar1.jpg":"avatars_videos/avatar1.mp4",
+                    "avatars_images/avatar2.jpg":"avatars_videos/avatar2.mp4",
+                    "avatars_images/avatar3.png":"avatars_videos/avatar3.mp4"
+                              }
+def streamlit_look():
+    """
+    Modest front-end code:)
+    """
+    data={}
+    st.title("Welcome to AI Lip Sync :)")
+    
+    # Add a brief app description
+    st.markdown("""
+    This app uses AI to synchronize a person's lip movements with any audio file. 
+    You can choose from built-in avatars or upload your own image/video, then provide audio 
+    to create realistic lip-synced videos. Powered by Wav2Lip and optimized for Apple Silicon.
+    """)
+    
+    # Add a guidelines section with an expander for best practices
+    with st.expander("📋 Guidelines & Best Practices (Click to expand)", expanded=False):
+        st.markdown("""
+        ### Guidelines for Best Results
+
+        #### Audio and Video Length
+        - Audio and video don't need to be exactly the same length
+        - If audio is shorter than video: Only the matching portion will be lip-synced
+        - If audio is longer than video: Audio will be trimmed to match video length
+
+        #### Face Quality
+        - Clear, well-lit frontal views of faces work best
+        - Faces should take up a reasonable portion of the frame
+        - Avoid extreme angles, heavy shadows, or partial face views
+
+        #### Audio Quality
+        - Clear speech with minimal background noise works best
+        - Consistent audio volume improves synchronization
+        - Supported formats: WAV, MP3
+
+        #### Video Quality
+        - Stable videos with minimal camera movement
+        - The person's mouth should be clearly visible
+        - Videos at 480p or higher resolution work best
+        - Very high-resolution videos will be automatically downscaled
+
+        #### Processing Tips
+        - Shorter videos process faster and often give better results
+        - "Fast animation" only moves the lips (quicker processing)
+        - "Slow animation" animates the full face (better quality, slower)
+        - Your M2 Max GPU will significantly speed up processing
+        """)
+    
+    # Option to choose between built-in avatars or upload a custom one
+    avatar_source = st.radio("Choose avatar source:", ["Upload my own image/video", "Use built-in avatars"])
+    
+    if avatar_source == "Use built-in avatars":
+        st.write("Please choose your avatar from the following options:")
+        avatar_img = image_select("", 
+                                ["avatars_images/avatar1.jpg",
+                                "avatars_images/avatar2.jpg",
+                                "avatars_images/avatar3.png",
+                                        ])
+        data["imge_path"] = avatar_img
+    else:
+        st.write("Upload an image or video file for your avatar:")
+        uploaded_file = st.file_uploader("Choose an image or video file", type=["jpg", "jpeg", "png", "mp4"], key="avatar_uploader")
+        
+        if uploaded_file is not None:
+            # Save the uploaded file
+            file_path = os.path.join("uploads", uploaded_file.name)
+            os.makedirs("uploads", exist_ok=True)
+            
+            with open(file_path, "wb") as f:
+                f.write(uploaded_file.getvalue())
+            
+            # Set the file path as image path
+            data["imge_path"] = file_path
+            st.success(f"File uploaded successfully: {uploaded_file.name}")
+            
+            # Preview the uploaded image/video
+            if uploaded_file.name.endswith(('.jpg', '.jpeg', '.png')):
+                st.image(file_path, caption="Uploaded Image")
+            elif uploaded_file.name.endswith('.mp4'):
+                st.video(file_path)
+                
+                # Analyze video quality for MP4 files
+                with st.spinner("Analyzing video quality..."):
+                    video_analysis = analyze_video_quality(file_path)
+                
+                # Display video quality analysis in a nice box
+                with st.expander("📊 Video Quality Analysis", expanded=True):
+                    col1, col2 = st.columns(2)
+                    
+                    with col1:
+                        st.markdown(f"**Resolution:** {video_analysis['resolution']}")
+                        st.markdown(f"**FPS:** {video_analysis['fps']}")
+                        st.markdown(f"**Duration:** {video_analysis['duration']}")
+                    
+                    with col2:
+                        quality_color = {
+                            "Excellent": "green",
+                            "Good": "lightgreen",
+                            "Fair": "orange",
+                            "Poor": "red",
+                            "Error": "red"
+                        }.get(video_analysis['quality'], "gray")
+                        
+                        st.markdown(f"**Quality:** <span style='color:{quality_color};font-weight:bold'>{video_analysis['quality']}</span>", unsafe_allow_html=True)
+                        st.markdown(f"**Face Detection:** {'✅ Detected' if video_analysis.get('face_detected', False) else '❌ Not detected'}")
+                    
+                    # Display the recommendation
+                    st.info(video_analysis['message'])
+    
+    # Option to choose between mic recording or upload audio file
+    audio_source = st.radio("Choose audio source:", ["Upload audio file", "Record with microphone"])
+    
+    if audio_source == "Record with microphone":
+        audio = mic_recorder(
+            start_prompt="Start recording",
+            stop_prompt="Stop recording", 
+            just_once=False,
+            use_container_width=False,
+            callback=None,
+            args=(),
+            kwargs={},
+            key=None)
+        
+        if audio:
+            st.audio(audio["bytes"])
+            data["audio"] = audio["bytes"]
+    else:
+        st.write("Upload an audio file:")
+        uploaded_audio = st.file_uploader("Choose an audio file", type=["wav", "mp3"], key="audio_uploader")
+        
+        if uploaded_audio is not None:
+            # Save the uploaded audio file
+            audio_path = os.path.join("uploads", uploaded_audio.name)
+            os.makedirs("uploads", exist_ok=True)
+            
+            with open(audio_path, "wb") as f:
+                f.write(uploaded_audio.getvalue())
+            
+            # Preview the uploaded audio
+            st.audio(audio_path)
+            
+            # Read the file into bytes for consistency with microphone recording
+            with open(audio_path, "rb") as f:
+                audio_bytes = f.read()
+            
+            data["audio"] = audio_bytes
+            st.success(f"Audio file uploaded successfully: {uploaded_audio.name}")
+            
+            # Analyze audio quality
+            with st.spinner("Analyzing audio quality..."):
+                audio_analysis = analyze_audio_quality(audio_path)
+            
+            # Display audio quality analysis in a nice box
+            with st.expander("🎵 Audio Quality Analysis", expanded=True):
+                col1, col2 = st.columns(2)
+                
+                with col1:
+                    st.markdown(f"**Duration:** {audio_analysis['duration']}")
+                    st.markdown(f"**Volume:** {audio_analysis['volume_quality']}")
+                
+                with col2:
+                    quality_color = {
+                        "Excellent": "green",
+                        "Good": "lightgreen",
+                        "Fair": "orange",
+                        "Poor": "red",
+                        "Error": "red"
+                    }.get(audio_analysis['quality'], "gray")
+                    
+                    st.markdown(f"**Quality:** <span style='color:{quality_color};font-weight:bold'>{audio_analysis['quality']}</span>", unsafe_allow_html=True)
+                    st.markdown(f"**Speech:** {audio_analysis['speech_quality']}")
+                
+                # Display the recommendation
+                st.info(audio_analysis['message'])
+    
+    return data
+
+def main():  
+    # Initialize session state to track processing status
+    if 'processed' not in st.session_state:
+        st.session_state.processed = False
+        
+    data = streamlit_look()
+    
+    # Add debug information
+    st.write("Debug info:")
+    if "imge_path" in data:
+        st.write(f"Image/Video path: {data['imge_path']}")
+    else:
+        st.write("No image/video selected yet")
+        
+    if "audio" in data:
+        st.write("Audio file selected ✓")
+    else:
+        st.write("No audio selected yet")
+    
+    # Only proceed if we have both image/video and audio data
+    if "imge_path" in data and "audio" in data:
+        st.write("This app will automatically save your audio when you click animate.")
+        save_record = st.button("save record manually")
+        st.write("With fast animation only the lips of the avatar will move, and it will take probably less than a minute for a record of about 30 seconds, but with slow animation choice, the full face of the avatar will move and it will take about 30 minutes for a record of about 30 seconds to get ready.")
+        model = load_model("wav2lip_checkpoints/wav2lip_gan.pth")
+        
+        # Check for duration mismatches between video and audio
+        if data["imge_path"].endswith('.mp4'):
+            # Save audio to temp file for analysis
+            if not os.path.exists('record.wav'):
+                with open('record.wav', mode='wb') as f:
+                    f.write(data["audio"])
+            
+            # Get durations
+            video_duration = get_video_duration(data["imge_path"])
+            audio_duration = get_audio_duration('record.wav')
+            
+            # Check for significant duration mismatch (more than 2 seconds difference)
+            if abs(video_duration - audio_duration) > 2:
+                st.warning(f"⚠️ Duration mismatch detected: Video is {video_duration:.1f}s and Audio is {audio_duration:.1f}s")
+                
+                # Create a tab for handling duration mismatches
+                with st.expander("Duration Mismatch Options (Click to expand)", expanded=True):
+                    st.info("The video and audio have different durations. Choose an option below:")
+                    
+                    if video_duration > audio_duration:
+                        if st.button("Trim Video to Match Audio Duration"):
+                            # Update duration values to match
+                            output_path = 'uploads/trimmed_input_video.mp4'
+                            with st.spinner(f"Trimming video from {video_duration:.1f}s to {audio_duration:.1f}s..."):
+                                success = trim_video(data["imge_path"], output_path, 0, audio_duration)
+                                
+                            if success:
+                                st.success("Video trimmed to match audio duration!")
+                                # Update the image path to use the trimmed video
+                                data["imge_path"] = output_path
+                                st.video(output_path)
+                    else:  # audio_duration > video_duration
+                        if st.button("Trim Audio to Match Video Duration"):
+                            # Update duration values to match
+                            output_path = 'uploads/trimmed_input_audio.wav'
+                            with st.spinner(f"Trimming audio from {audio_duration:.1f}s to {video_duration:.1f}s..."):
+                                success = trim_audio('record.wav', output_path, 0, video_duration)
+                                
+                            if success:
+                                st.success("Audio trimmed to match video duration!")
+                                # Update the audio data with the trimmed audio
+                                with open(output_path, "rb") as f:
+                                    data["audio"] = f.read()
+                                # Save the trimmed audio as record.wav
+                                with open('record.wav', mode='wb') as f:
+                                    f.write(data["audio"])
+                                st.audio(output_path)
+        
+        # Animation buttons
+        fast_animate = st.button("fast animate")
+        slower_animate = st.button("slower animate")
+        
+        # Function to save the audio record
+        def save_audio_record():
+            if os.path.exists('record.wav'):
+                os.remove('record.wav')
+            with open('record.wav', mode='wb') as f:
+                f.write(data["audio"])
+            st.write("Audio record saved!")
+        
+        if save_record:
+            save_audio_record()
+        
+        # Show previously generated results if they exist and we're not generating new ones
+        if os.path.exists('wav2lip/results/result_voice.mp4') and st.session_state.processed and not (fast_animate or slower_animate):
+            st.video('wav2lip/results/result_voice.mp4')
+            display_trim_options('wav2lip/results/result_voice.mp4')
+        
+        if fast_animate:
+            # Automatically save the record before animation
+            save_audio_record()
+            
+            progress_placeholder = st.empty()
+            status_placeholder = st.empty()
+            
+            progress_bar = progress_placeholder.progress(0, text="Processing: 0% complete")
+            status_placeholder.info("Preparing to process...")
+            
+            # Call the inference function inside a try block with progress updates at key points
+            try:
+                # Initialize a progress tracker
+                progress_steps = [
+                    (0, "Starting processing..."),
+                    (15, "Step 1/4: Loading and analyzing video frames"),
+                    (30, "Step 2/4: Performing face detection (this may take a while for long videos)"),
+                    (60, "Step 3/4: Generating lip-synced frames"),
+                    (80, "Step 4/4: Creating final video with audio"),
+                    (100, "Processing complete!")
+                ]
+                current_step = 0
+                
+                # Redirect stdout to capture progress information
+                import io
+                sys.stdout = io.StringIO()
+                
+                # Update progress for the initial step
+                progress, message = progress_steps[current_step]
+                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                status_placeholder.info(message)
+                current_step += 1
+                
+                # Run the inference in a background thread
+                with concurrent.futures.ThreadPoolExecutor() as executor:
+                    # Start the inference process
+                    future = executor.submit(inference.main, data["imge_path"], "record.wav", model)
+                    
+                    # Monitor the output for progress indicators
+                    while not future.done():
+                        captured_output = sys.stdout.getvalue()
+                        
+                        # Check for progress indicators and update UI
+                        if current_step < len(progress_steps):
+                            # Check for stage 1 completion: frames read
+                            if current_step == 1 and "Number of frames available for inference" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                            # Check for stage 2 completion: face detection
+                            elif current_step == 2 and "Face detection completed successfully" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                            # Check for stage 3 completion: ffmpeg started
+                            elif current_step == 3 and "ffmpeg" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                        
+                        # Sleep to avoid excessive CPU usage
+                        time.sleep(0.5)
+                    
+                    try:
+                        # Get the result or propagate exceptions
+                        future.result()
+                        
+                        # Show completion
+                        progress, message = progress_steps[-1]
+                        progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                        status_placeholder.success("Lip sync complete! Your video is ready.")
+                    except Exception as e:
+                        raise e
+                
+                # Restore stdout
+                sys.stdout = sys.__stdout__
+                
+                if os.path.exists('wav2lip/results/result_voice.mp4'):
+                    st.video('wav2lip/results/result_voice.mp4')
+                    display_trim_options('wav2lip/results/result_voice.mp4')
+                    # Set processed flag to True after successful processing
+                    st.session_state.processed = True
+                    
+            except Exception as e:
+                # Restore stdout in case of error
+                sys.stdout = sys.__stdout__
+                
+                progress_placeholder.empty()
+                status_placeholder.error(f"Error during processing: {str(e)}")
+                st.error("Failed to generate video. Please try again or use a different image/audio.")
+        
+        if slower_animate:
+            # Automatically save the record before animation
+            save_audio_record()
+            
+            progress_placeholder = st.empty()
+            status_placeholder = st.empty()
+            
+            progress_bar = progress_placeholder.progress(0, text="Processing: 0% complete")
+            status_placeholder.info("Preparing to process...")
+            
+            # Derive the video path from the selected avatar
+            if data["imge_path"].endswith('.mp4'):
+                video_path = data["imge_path"]
+            else:
+                # Get the avatar video path for the selected avatar
+                avatar_list = load_avatar_videos_for_slow_animation("./data/avatars/samples")
+                video_path = avatar_list[available_avatars_for_slow.index(avatar_choice)]
+            
+            try:
+                # Initialize a progress tracker
+                progress_steps = [
+                    (0, "Starting processing..."),
+                    (15, "Step 1/4: Loading and analyzing video frames"),
+                    (30, "Step 2/4: Performing face detection (this may take a while for long videos)"),
+                    (60, "Step 3/4: Generating lip-synced frames with full-face animation"),
+                    (80, "Step 4/4: Creating final video with audio"),
+                    (100, "Processing complete!")
+                ]
+                current_step = 0
+                
+                # Redirect stdout to capture progress information
+                import io
+                sys.stdout = io.StringIO()
+                
+                # Update progress for the initial step
+                progress, message = progress_steps[current_step]
+                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                status_placeholder.info(message)
+                current_step += 1
+                
+                # Run the inference in a background thread
+                with concurrent.futures.ThreadPoolExecutor() as executor:
+                    # Start the inference process
+                    future = executor.submit(inference.main, video_path, "record.wav", model, slow_mode=True)
+                    
+                    # Monitor the output for progress indicators
+                    while not future.done():
+                        captured_output = sys.stdout.getvalue()
+                        
+                        # Check for progress indicators and update UI
+                        if current_step < len(progress_steps):
+                            # Check for stage 1 completion: frames read
+                            if current_step == 1 and "Number of frames available for inference" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                            # Check for stage 2 completion: face detection
+                            elif current_step == 2 and "Face detection completed successfully" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                            # Check for stage 3 completion: ffmpeg started
+                            elif current_step == 3 and "ffmpeg" in captured_output:
+                                progress, message = progress_steps[current_step]
+                                progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                                status_placeholder.info(message)
+                                current_step += 1
+                        
+                        # Sleep to avoid excessive CPU usage
+                        time.sleep(0.5)
+                    
+                    try:
+                        # Get the result or propagate exceptions
+                        future.result()
+                        
+                        # Show completion
+                        progress, message = progress_steps[-1]
+                        progress_bar.progress(progress, text=f"Processing: {progress}% complete")
+                        status_placeholder.success("Lip sync complete! Your video is ready.")
+                    except Exception as e:
+                        raise e
+                
+                # Restore stdout
+                sys.stdout = sys.__stdout__
+                
+                if os.path.exists('wav2lip/results/result_voice.mp4'):
+                    st.video('wav2lip/results/result_voice.mp4')
+                    display_trim_options('wav2lip/results/result_voice.mp4')
+                    # Set processed flag to True after successful processing
+                    st.session_state.processed = True
+            except Exception as e:
+                # Restore stdout in case of error
+                sys.stdout = sys.__stdout__
+                
+                progress_placeholder.empty()
+                status_placeholder.error(f"Error during processing: {str(e)}")
+                st.error("Failed to generate video. Please try again or use a different video/audio.")
+    else:
+        if "imge_path" not in data and "audio" not in data:
+            st.warning("Please upload both an image/video AND provide audio to continue.")
+        elif "imge_path" not in data:
+            st.warning("Please select or upload an image/video to continue.")
+        else:
+            st.warning("Please provide audio to continue.")
+
+# Function to display trim options and handle video trimming
+def display_trim_options(video_path):
+    """Display options to trim the video and handle the trimming process"""
+    st.subheader("Video Processing Options")
+    
+    # Check if the video exists first
+    if not os.path.exists(video_path):
+        st.error(f"Video file not found at {video_path}. Try running the animation again.")
+        return
+    
+    # Add tabs for different operations
+    download_tab, trim_tab = st.tabs(["Download Original", "Trim Video"])
+    
+    with download_tab:
+        st.write("Download the original generated video:")
+        try:
+            st.video(video_path)
+            st.download_button(
+                label="Download Original Video",
+                data=open(video_path, 'rb').read(),
+                file_name="original_lip_sync_video.mp4",
+                mime="video/mp4"
+            )
+        except Exception as e:
+            st.error(f"Error loading video: {str(e)}")
+    
+    with trim_tab:
+        st.write("You can trim the generated video to remove unwanted parts from the beginning or end.")
+        
+        duration = get_video_duration(video_path)
+        if duration <= 0:
+            st.error("Could not determine video duration")
+            return
+        
+        # Display video duration
+        st.write(f"Video duration: {duration:.2f} seconds")
+        
+        # Create a slider for selecting start and end times
+        col1, col2 = st.columns(2)
+        
+        with col1:
+            start_time = st.slider("Start time (seconds)", 
+                                min_value=0.0, 
+                                max_value=float(duration), 
+                                value=0.0,
+                                step=0.1)
+            st.write(f"Start at: {start_time:.1f}s")
+        
+        with col2:
+            end_time = st.slider("End time (seconds)", 
+                                min_value=0.0, 
+                                max_value=float(duration), 
+                                value=float(duration),
+                                step=0.1)
+            st.write(f"End at: {end_time:.1f}s")
+        
+        # Display trim duration
+        trim_duration = end_time - start_time
+        st.info(f"Trimmed video duration will be: {trim_duration:.1f} seconds")
+        
+        # Validate the selected range
+        if start_time >= end_time:
+            st.error("Start time must be less than end time")
+            return
+        
+        # Button to perform trimming
+        if st.button("Trim Video"):
+            # Generate output path
+            output_path = 'wav2lip/results/trimmed_video.mp4'
+            
+            # Show progress
+            with st.spinner("Trimming video..."):
+                success = trim_video(video_path, output_path, start_time, end_time)
+            
+            if success:
+                st.success("Video trimmed successfully!")
+                try:
+                    st.video(output_path)
+                    
+                    # Add download button for trimmed video
+                    st.download_button(
+                        label="Download Trimmed Video",
+                        data=open(output_path, 'rb').read(),
+                        file_name="trimmed_lip_sync_video.mp4",
+                        mime="video/mp4"
+                    )
+                except Exception as e:
+                    st.error(f"Error displaying trimmed video: {str(e)}")
+            else:
+                st.error("Failed to trim video. Try again with different timing parameters.")
+
+# Function to trim video using ffmpeg
+def trim_video(input_path, output_path, start_time, end_time):
+    """
+    Trim a video using ffmpeg from start_time to end_time.
+    
+    Args:
+        input_path: Path to the input video
+        output_path: Path to save the trimmed video
+        start_time: Start time in seconds
+        end_time: End time in seconds
+    
+    Returns:
+        bool: True if successful, False otherwise
+    """
+    try:
+        # Check if input file exists
+        if not os.path.exists(input_path):
+            st.error(f"Input video not found: {input_path}")
+            return False
+            
+        # Format the command - use -ss before -i for faster seeking
+        # Add quotes around file paths to handle spaces and special characters
+        command = f'ffmpeg -y -ss {start_time} -i "{input_path}" -to {end_time} -c:v copy -c:a copy "{output_path}"'
+        
+        # Use subprocess.run for better error handling
+        result = subprocess.run(
+            command, 
+            shell=True, 
+            stdout=subprocess.PIPE, 
+            stderr=subprocess.PIPE,
+            text=True
+        )
+        
+        if result.returncode != 0:
+            st.error(f"FFMPEG error: {result.stderr}")
+            return False
+            
+        # Verify the output file exists and has a size greater than 0
+        if os.path.exists(output_path) and os.path.getsize(output_path) > 0:
+            return True
+        else:
+            st.error("Output file was not created correctly")
+            return False
+            
+    except Exception as e:
+        st.error(f"Error trimming video: {str(e)}")
+        return False
+
+# Function to get video duration
+def get_video_duration(video_path):
+    """Get the duration of a video file in seconds"""
+    try:
+        video = cv2.VideoCapture(video_path)
+        fps = video.get(cv2.CAP_PROP_FPS)
+        frame_count = int(video.get(cv2.CAP_PROP_FRAME_COUNT))
+        video.release()
+        
+        duration = frame_count / fps if fps > 0 else 0
+        return duration
+    except Exception as e:
+        st.error(f"Error getting video duration: {str(e)}")
+        return 0
+
+# Function to get audio duration
+def get_audio_duration(audio_path):
+    """Get the duration of an audio file in seconds"""
+    try:
+        y, sr = librosa.load(audio_path, sr=None)
+        duration = librosa.get_duration(y=y, sr=sr)
+        return duration
+    except Exception as e:
+        st.error(f"Error getting audio duration: {str(e)}")
+        return 0
+
+# Function to trim audio file
+def trim_audio(input_path, output_path, start_time, end_time):
+    """Trim an audio file to the specified start and end times"""
+    try:
+        # Command to trim audio using ffmpeg
+        command = f'ffmpeg -y -i "{input_path}" -ss {start_time} -to {end_time} -c copy "{output_path}"'
+        
+        # Execute the command
+        subprocess.call(command, shell=True)
+        
+        # Check if output file exists
+        if os.path.exists(output_path):
+            return True
+        else:
+            st.error("Output audio file was not created correctly")
+            return False
+            
+    except Exception as e:
+        st.error(f"Error trimming audio: {str(e)}")
+        return False
+
+if __name__ == "__main__":
+    main()

packages.txt

@@ -0,0 +1,3 @@
+python3-opencv
+libgl1-mesa-dev
+ffmpeg

requirements.txt

@@ -0,0 +1,12 @@
+numpy==1.26.3
+scipy==1.12.0
+iou==0.1.0
+librosa==0.10.1
+opencv_contrib_python==4.9.0.80
+streamlit==1.31.0 
+streamlit_image_select==0.6.0
+streamlit_mic_recorder==0.0.4
+torch==2.2.1
+tqdm==4.64.1
+gdown
+matplotlib==3.10.1

wav2lip/audio.py

@@ -0,0 +1,136 @@
+import librosa
+import librosa.filters
+import numpy as np
+# import tensorflow as tf
+from scipy import signal
+from scipy.io import wavfile
+from .hparams import hparams as hp
+
+def load_wav(path, sr):
+    return librosa.core.load(path, sr=sr)[0]
+
+def save_wav(wav, path, sr):
+    wav *= 32767 / max(0.01, np.max(np.abs(wav)))
+    #proposed by @dsmiller
+    wavfile.write(path, sr, wav.astype(np.int16))
+
+def save_wavenet_wav(wav, path, sr):
+    librosa.output.write_wav(path, wav, sr=sr)
+
+def preemphasis(wav, k, preemphasize=True):
+    if preemphasize:
+        return signal.lfilter([1, -k], [1], wav)
+    return wav
+
+def inv_preemphasis(wav, k, inv_preemphasize=True):
+    if inv_preemphasize:
+        return signal.lfilter([1], [1, -k], wav)
+    return wav
+
+def get_hop_size():
+    hop_size = hp.hop_size
+    if hop_size is None:
+        assert hp.frame_shift_ms is not None
+        hop_size = int(hp.frame_shift_ms / 1000 * hp.sample_rate)
+    return hop_size
+
+def linearspectrogram(wav):
+    D = _stft(preemphasis(wav, hp.preemphasis, hp.preemphasize))
+    S = _amp_to_db(np.abs(D)) - hp.ref_level_db
+    
+    if hp.signal_normalization:
+        return _normalize(S)
+    return S
+
+def melspectrogram(wav):
+    D = _stft(preemphasis(wav, hp.preemphasis, hp.preemphasize))
+    S = _amp_to_db(_linear_to_mel(np.abs(D))) - hp.ref_level_db
+    
+    if hp.signal_normalization:
+        return _normalize(S)
+    return S
+
+def _lws_processor():
+    import lws
+    return lws.lws(hp.n_fft, get_hop_size(), fftsize=hp.win_size, mode="speech")
+
+def _stft(y):
+    if hp.use_lws:
+        return _lws_processor(hp).stft(y).T
+    else:
+        return librosa.stft(y=y, n_fft=hp.n_fft, hop_length=get_hop_size(), win_length=hp.win_size)
+
+##########################################################
+#Those are only correct when using lws!!! (This was messing with Wavenet quality for a long time!)
+def num_frames(length, fsize, fshift):
+    """Compute number of time frames of spectrogram
+    """
+    pad = (fsize - fshift)
+    if length % fshift == 0:
+        M = (length + pad * 2 - fsize) // fshift + 1
+    else:
+        M = (length + pad * 2 - fsize) // fshift + 2
+    return M
+
+
+def pad_lr(x, fsize, fshift):
+    """Compute left and right padding
+    """
+    M = num_frames(len(x), fsize, fshift)
+    pad = (fsize - fshift)
+    T = len(x) + 2 * pad
+    r = (M - 1) * fshift + fsize - T
+    return pad, pad + r
+##########################################################
+#Librosa correct padding
+def librosa_pad_lr(x, fsize, fshift):
+    return 0, (x.shape[0] // fshift + 1) * fshift - x.shape[0]
+
+# Conversions
+_mel_basis = None
+
+def _linear_to_mel(spectogram):
+    global _mel_basis
+    if _mel_basis is None:
+        _mel_basis = _build_mel_basis()
+    return np.dot(_mel_basis, spectogram)
+
+
+def _build_mel_basis():
+    assert hp.fmax <= hp.sample_rate // 2
+    return librosa.filters.mel(sr=hp.sample_rate, n_fft=hp.n_fft, n_mels=hp.num_mels,
+                               fmin=hp.fmin, fmax=hp.fmax)
+def _amp_to_db(x):
+    min_level = np.exp(hp.min_level_db / 20 * np.log(10))
+    return 20 * np.log10(np.maximum(min_level, x))
+
+def _db_to_amp(x):
+    return np.power(10.0, (x) * 0.05)
+
+def _normalize(S):
+    if hp.allow_clipping_in_normalization:
+        if hp.symmetric_mels:
+            return np.clip((2 * hp.max_abs_value) * ((S - hp.min_level_db) / (-hp.min_level_db)) - hp.max_abs_value,
+                           -hp.max_abs_value, hp.max_abs_value)
+        else:
+            return np.clip(hp.max_abs_value * ((S - hp.min_level_db) / (-hp.min_level_db)), 0, hp.max_abs_value)
+    
+    assert S.max() <= 0 and S.min() - hp.min_level_db >= 0
+    if hp.symmetric_mels:
+        return (2 * hp.max_abs_value) * ((S - hp.min_level_db) / (-hp.min_level_db)) - hp.max_abs_value
+    else:
+        return hp.max_abs_value * ((S - hp.min_level_db) / (-hp.min_level_db))
+
+def _denormalize(D):
+    if hp.allow_clipping_in_normalization:
+        if hp.symmetric_mels:
+            return (((np.clip(D, -hp.max_abs_value,
+                              hp.max_abs_value) + hp.max_abs_value) * -hp.min_level_db / (2 * hp.max_abs_value))
+                    + hp.min_level_db)
+        else:
+            return ((np.clip(D, 0, hp.max_abs_value) * -hp.min_level_db / hp.max_abs_value) + hp.min_level_db)
+    
+    if hp.symmetric_mels:
+        return (((D + hp.max_abs_value) * -hp.min_level_db / (2 * hp.max_abs_value)) + hp.min_level_db)
+    else:
+        return ((D * -hp.min_level_db / hp.max_abs_value) + hp.min_level_db)

wav2lip/face_detection/README.md

@@ -0,0 +1 @@
+The code for Face Detection in this folder has been taken from the wonderful [face_alignment](https://github.com/1adrianb/face-alignment) repository. This has been modified to take batches of faces at a time. 

wav2lip/face_detection/__init__.py

@@ -0,0 +1,7 @@
+# -*- coding: utf-8 -*-
+
+__author__ = """Adrian Bulat"""
+__email__ = '[email protected]'
+__version__ = '1.0.1'
+
+from .api import FaceAlignment, LandmarksType, NetworkSize

wav2lip/face_detection/api.py

@@ -0,0 +1,119 @@
+from __future__ import print_function
+import os
+import torch
+from torch.utils.model_zoo import load_url
+from enum import Enum
+import numpy as np
+import cv2
+from .detection import sfd
+try:
+    import urllib.request as request_file
+except BaseException:
+    import urllib as request_file
+
+from .models import FAN, ResNetDepth
+from .utils import *
+
+
+class LandmarksType(Enum):
+    """Enum class defining the type of landmarks to detect.
+
+    ``_2D`` - the detected points ``(x,y)`` are detected in a 2D space and follow the visible contour of the face
+    ``_2halfD`` - this points represent the projection of the 3D points into 3D
+    ``_3D`` - detect the points ``(x,y,z)``` in a 3D space
+
+    """
+    _2D = 1
+    _2halfD = 2
+    _3D = 3
+
+
+class NetworkSize(Enum):
+    # TINY = 1
+    # SMALL = 2
+    # MEDIUM = 3
+    LARGE = 4
+
+    def __new__(cls, value):
+        member = object.__new__(cls)
+        member._value_ = value
+        return member
+
+    def __int__(self):
+        return self.value
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+
+class FaceAlignment:
+    def __init__(self, landmarks_type, network_size=NetworkSize.LARGE,
+                 device='cuda', flip_input=False, face_detector='sfd', verbose=False):
+        self.device = device
+        self.flip_input = flip_input
+        self.landmarks_type = landmarks_type
+        self.verbose = verbose
+
+        network_size = int(network_size)
+
+        if 'cuda' in device or 'mps' in device:
+            torch.backends.cudnn.benchmark = True
+            if 'mps' in device and verbose:
+                print("Using Apple Silicon GPU (MPS) for face detection.")
+
+        # Get the face detector
+        #face_detector_module = __import__('from .detection. import' + face_detector,
+        #                                  globals(), locals(), [face_detector], 0)
+        #self.face_detector = face_detector_module.FaceDetector(device=device, verbose=verbose)
+        try:
+            self.face_detector = sfd.FaceDetector(device=device, verbose=verbose)
+        except Exception as e:
+            print(f"Error initializing face detector: {e}")
+            print("Falling back to CPU for face detection.")
+            # If detection fails on GPU (MPS/CUDA), fall back to CPU
+            self.device = 'cpu'
+            self.face_detector = sfd.FaceDetector(device='cpu', verbose=verbose)
+
+    def get_detections_for_batch(self, images):
+        """
+        Returns a list of bounding boxes for each image in the batch.
+        If no face is detected, returns None for that image.
+        """
+        try:
+            # Convert to RGB for face detection 
+            images = images.copy()
+            if images.shape[-1] == 3:
+                images = images[..., ::-1]  # BGR to RGB
+            
+            # Get face detections
+            detected_faces = self.face_detector.detect_from_batch(images)
+            
+            results = []
+            for i, d in enumerate(detected_faces):
+                if len(d) == 0:
+                    # No face detected
+                    results.append(None)
+                    continue
+                    
+                # Use the first (highest confidence) face
+                d = d[0]
+                # Ensure values are valid
+                d = np.clip(d, 0, None)
+                
+                # Extract coordinates
+                try:
+                    x1, y1, x2, y2 = map(int, d[:-1])
+                    # Sanity check on coordinates
+                    if x1 >= x2 or y1 >= y2 or x1 < 0 or y1 < 0:
+                        print(f"Invalid face coordinates: {(x1, y1, x2, y2)}")
+                        results.append(None)
+                    else:
+                        results.append((x1, y1, x2, y2))
+                except Exception as e:
+                    print(f"Error processing detection: {str(e)}")
+                    results.append(None)
+            
+            return results
+            
+        except Exception as e:
+            print(f"Error in batch face detection: {str(e)}")
+            # Return None for all images
+            return [None] * len(images)

wav2lip/face_detection/detection/__init__.py

@@ -0,0 +1 @@
+from .core import FaceDetector

wav2lip/face_detection/detection/core.py

@@ -0,0 +1,134 @@
+import logging
+import glob
+from tqdm import tqdm
+import numpy as np
+import torch
+import cv2
+
+
+class FaceDetector(object):
+    """An abstract class representing a face detector.
+
+    Any other face detection implementation must subclass it. All subclasses
+    must implement ``detect_from_image``, that return a list of detected
+    bounding boxes. Optionally, for speed considerations detect from path is
+    recommended.
+    """
+
+    def __init__(self, device, verbose):
+        self.device = device
+        self.verbose = verbose
+
+        if verbose:
+            if 'cpu' in device:
+                logger = logging.getLogger(__name__)
+                logger.warning("Detection running on CPU, this may be potentially slow.")
+            elif 'mps' in device:
+                logger = logging.getLogger(__name__)
+                logger.info("Detection running on Apple Silicon GPU (MPS).")
+
+        if 'cpu' not in device and 'cuda' not in device and 'mps' not in device:
+            logger = logging.getLogger(__name__)
+            if verbose:
+                logger.error("Expected values for device are: {cpu, cuda, mps} but got: %s", device)
+            raise ValueError(f"Invalid device type: {device}. Expected 'cpu', 'cuda', or 'mps'.")
+
+    def detect_from_image(self, tensor_or_path):
+        """Detects faces in a given image.
+
+        This function detects the faces present in a provided BGR(usually)
+        image. The input can be either the image itself or the path to it.
+
+        Arguments:
+            tensor_or_path {numpy.ndarray, torch.tensor or string} -- the path
+            to an image or the image itself.
+
+        Example::
+
+            >>> path_to_image = 'data/image_01.jpg'
+            ...   detected_faces = detect_from_image(path_to_image)
+            [A list of bounding boxes (x1, y1, x2, y2)]
+            >>> image = cv2.imread(path_to_image)
+            ...   detected_faces = detect_from_image(image)
+            [A list of bounding boxes (x1, y1, x2, y2)]
+
+        """
+        raise NotImplementedError
+
+    def detect_from_directory(self, path, extensions=['.jpg', '.png'], recursive=False, show_progress_bar=True):
+        """Detects faces from all the images present in a given directory.
+
+        Arguments:
+            path {string} -- a string containing a path that points to the folder containing the images
+
+        Keyword Arguments:
+            extensions {list} -- list of string containing the extensions to be
+            consider in the following format: ``.extension_name`` (default:
+            {['.jpg', '.png']}) recursive {bool} -- option wherever to scan the
+            folder recursively (default: {False}) show_progress_bar {bool} --
+            display a progressbar (default: {True})
+
+        Example:
+        >>> directory = 'data'
+        ...   detected_faces = detect_from_directory(directory)
+        {A dictionary of [lists containing bounding boxes(x1, y1, x2, y2)]}
+
+        """
+        if self.verbose:
+            logger = logging.getLogger(__name__)
+
+        if len(extensions) == 0:
+            if self.verbose:
+                logger.error("Expected at list one extension, but none was received.")
+            raise ValueError
+
+        if self.verbose:
+            logger.info("Constructing the list of images.")
+        additional_pattern = '/**/*' if recursive else '/*'
+        files = []
+        for extension in extensions:
+            files.extend(glob.glob(path + additional_pattern + extension, recursive=recursive))
+
+        if self.verbose:
+            logger.info("Finished searching for images. %s images found", len(files))
+            logger.info("Preparing to run the detection.")
+
+        predictions = {}
+        for image_path in tqdm(files, disable=not show_progress_bar):
+            if self.verbose:
+                logger.info("Running the face detector on image: %s", image_path)
+            predictions[image_path] = self.detect_from_image(image_path)
+
+        if self.verbose:
+            logger.info("The detector was successfully run on all %s images", len(files))
+
+        return predictions
+
+    @property
+    def reference_scale(self):
+        raise NotImplementedError
+
+    @property
+    def reference_x_shift(self):
+        raise NotImplementedError
+
+    @property
+    def reference_y_shift(self):
+        raise NotImplementedError
+
+    @staticmethod
+    def tensor_or_path_to_ndarray(tensor_or_path, rgb=True):
+        """Convert path (represented as a string) or torch.tensor to a numpy.ndarray
+
+        Arguments:
+            tensor_or_path {numpy.ndarray, torch.tensor or string} -- path to the image, or the image itself
+        """
+        if isinstance(tensor_or_path, str):
+            return cv2.imread(tensor_or_path) if not rgb else cv2.imread(tensor_or_path)[..., ::-1]
+        elif torch.is_tensor(tensor_or_path):
+            # Call cpu in case its coming from cuda
+            return tensor_or_path.cpu().numpy()[..., ::-1].copy() if not rgb else tensor_or_path.cpu().numpy()
+        elif isinstance(tensor_or_path, np.ndarray):
+            return tensor_or_path[..., ::-1].copy() if not rgb else tensor_or_path
+        else:
+            raise TypeError

wav2lip/face_detection/detection/sfd/__init__.py

@@ -0,0 +1 @@
+from .sfd_detector import SFDDetector as FaceDetector

wav2lip/face_detection/detection/sfd/bbox.py

@@ -0,0 +1,129 @@
+from __future__ import print_function
+import os
+import sys
+import cv2
+import random
+import datetime
+import time
+import math
+import argparse
+import numpy as np
+import torch
+
+try:
+    from iou import IOU
+except BaseException:
+    # IOU cython speedup 10x
+    def IOU(ax1, ay1, ax2, ay2, bx1, by1, bx2, by2):
+        sa = abs((ax2 - ax1) * (ay2 - ay1))
+        sb = abs((bx2 - bx1) * (by2 - by1))
+        x1, y1 = max(ax1, bx1), max(ay1, by1)
+        x2, y2 = min(ax2, bx2), min(ay2, by2)
+        w = x2 - x1
+        h = y2 - y1
+        if w < 0 or h < 0:
+            return 0.0
+        else:
+            return 1.0 * w * h / (sa + sb - w * h)
+
+
+def bboxlog(x1, y1, x2, y2, axc, ayc, aww, ahh):
+    xc, yc, ww, hh = (x2 + x1) / 2, (y2 + y1) / 2, x2 - x1, y2 - y1
+    dx, dy = (xc - axc) / aww, (yc - ayc) / ahh
+    dw, dh = math.log(ww / aww), math.log(hh / ahh)
+    return dx, dy, dw, dh
+
+
+def bboxloginv(dx, dy, dw, dh, axc, ayc, aww, ahh):
+    xc, yc = dx * aww + axc, dy * ahh + ayc
+    ww, hh = math.exp(dw) * aww, math.exp(dh) * ahh
+    x1, x2, y1, y2 = xc - ww / 2, xc + ww / 2, yc - hh / 2, yc + hh / 2
+    return x1, y1, x2, y2
+
+
+def nms(dets, thresh):
+    if 0 == len(dets):
+        return []
+    x1, y1, x2, y2, scores = dets[:, 0], dets[:, 1], dets[:, 2], dets[:, 3], dets[:, 4]
+    areas = (x2 - x1 + 1) * (y2 - y1 + 1)
+    order = scores.argsort()[::-1]
+
+    keep = []
+    while order.size > 0:
+        i = order[0]
+        keep.append(i)
+        xx1, yy1 = np.maximum(x1[i], x1[order[1:]]), np.maximum(y1[i], y1[order[1:]])
+        xx2, yy2 = np.minimum(x2[i], x2[order[1:]]), np.minimum(y2[i], y2[order[1:]])
+
+        w, h = np.maximum(0.0, xx2 - xx1 + 1), np.maximum(0.0, yy2 - yy1 + 1)
+        ovr = w * h / (areas[i] + areas[order[1:]] - w * h)
+
+        inds = np.where(ovr <= thresh)[0]
+        order = order[inds + 1]
+
+    return keep
+
+
+def encode(matched, priors, variances):
+    """Encode the variances from the priorbox layers into the ground truth boxes
+    we have matched (based on jaccard overlap) with the prior boxes.
+    Args:
+        matched: (tensor) Coords of ground truth for each prior in point-form
+            Shape: [num_priors, 4].
+        priors: (tensor) Prior boxes in center-offset form
+            Shape: [num_priors,4].
+        variances: (list[float]) Variances of priorboxes
+    Return:
+        encoded boxes (tensor), Shape: [num_priors, 4]
+    """
+
+    # dist b/t match center and prior's center
+    g_cxcy = (matched[:, :2] + matched[:, 2:]) / 2 - priors[:, :2]
+    # encode variance
+    g_cxcy /= (variances[0] * priors[:, 2:])
+    # match wh / prior wh
+    g_wh = (matched[:, 2:] - matched[:, :2]) / priors[:, 2:]
+    g_wh = torch.log(g_wh) / variances[1]
+    # return target for smooth_l1_loss
+    return torch.cat([g_cxcy, g_wh], 1)  # [num_priors,4]
+
+
+def decode(loc, priors, variances):
+    """Decode locations from predictions using priors to undo
+    the encoding we did for offset regression at train time.
+    Args:
+        loc (tensor): location predictions for loc layers,
+            Shape: [num_priors,4]
+        priors (tensor): Prior boxes in center-offset form.
+            Shape: [num_priors,4].
+        variances: (list[float]) Variances of priorboxes
+    Return:
+        decoded bounding box predictions
+    """
+
+    boxes = torch.cat((
+        priors[:, :2] + loc[:, :2] * variances[0] * priors[:, 2:],
+        priors[:, 2:] * torch.exp(loc[:, 2:] * variances[1])), 1)
+    boxes[:, :2] -= boxes[:, 2:] / 2
+    boxes[:, 2:] += boxes[:, :2]
+    return boxes
+
+def batch_decode(loc, priors, variances):
+    """Decode locations from predictions using priors to undo
+    the encoding we did for offset regression at train time.
+    Args:
+        loc (tensor): location predictions for loc layers,
+            Shape: [num_priors,4]
+        priors (tensor): Prior boxes in center-offset form.
+            Shape: [num_priors,4].
+        variances: (list[float]) Variances of priorboxes
+    Return:
+        decoded bounding box predictions
+    """
+
+    boxes = torch.cat((
+        priors[:, :, :2] + loc[:, :, :2] * variances[0] * priors[:, :, 2:],
+        priors[:, :, 2:] * torch.exp(loc[:, :, 2:] * variances[1])), 2)
+    boxes[:, :, :2] -= boxes[:, :, 2:] / 2
+    boxes[:, :, 2:] += boxes[:, :, :2]
+    return boxes

wav2lip/face_detection/detection/sfd/detect.py

@@ -0,0 +1,112 @@
+import torch
+import torch.nn.functional as F
+
+import os
+import sys
+import cv2
+import random
+import datetime
+import math
+import argparse
+import numpy as np
+
+import scipy.io as sio
+import zipfile
+from .net_s3fd import s3fd
+from .bbox import *
+
+
+def detect(net, img, device):
+    img = img - np.array([104, 117, 123])
+    img = img.transpose(2, 0, 1)
+    img = img.reshape((1,) + img.shape)
+
+    if 'cuda' in device or 'mps' in device:
+        torch.backends.cudnn.benchmark = True
+
+    img = torch.from_numpy(img).float().to(device)
+    BB, CC, HH, WW = img.size()
+    with torch.no_grad():
+        olist = net(img)
+
+    bboxlist = []
+    for i in range(len(olist) // 2):
+        olist[i * 2] = F.softmax(olist[i * 2], dim=1)
+    olist = [oelem.data.cpu() for oelem in olist]
+    for i in range(len(olist) // 2):
+        ocls, oreg = olist[i * 2], olist[i * 2 + 1]
+        FB, FC, FH, FW = ocls.size()  # feature map size
+        stride = 2**(i + 2)    # 4,8,16,32,64,128
+        anchor = stride * 4
+        poss = zip(*np.where(ocls[:, 1, :, :] > 0.05))
+        for Iindex, hindex, windex in poss:
+            axc, ayc = stride / 2 + windex * stride, stride / 2 + hindex * stride
+            score = ocls[0, 1, hindex, windex]
+            loc = oreg[0, :, hindex, windex].contiguous().view(1, 4)
+            priors = torch.Tensor([[axc / 1.0, ayc / 1.0, stride * 4 / 1.0, stride * 4 / 1.0]])
+            variances = [0.1, 0.2]
+            box = decode(loc, priors, variances)
+            x1, y1, x2, y2 = box[0] * 1.0
+            # cv2.rectangle(imgshow,(int(x1),int(y1)),(int(x2),int(y2)),(0,0,255),1)
+            bboxlist.append([x1, y1, x2, y2, score])
+    bboxlist = np.array(bboxlist)
+    if 0 == len(bboxlist):
+        bboxlist = np.zeros((1, 5))
+
+    return bboxlist
+
+def batch_detect(net, imgs, device):
+    imgs = imgs - np.array([104, 117, 123])
+    imgs = imgs.transpose(0, 3, 1, 2)
+
+    if 'cuda' in device or 'mps' in device:
+        torch.backends.cudnn.benchmark = True
+
+    imgs = torch.from_numpy(imgs).float().to(device)
+    BB, CC, HH, WW = imgs.size()
+    with torch.no_grad():
+        olist = net(imgs)
+
+    bboxlist = []
+    for i in range(len(olist) // 2):
+        olist[i * 2] = F.softmax(olist[i * 2], dim=1)
+    olist = [oelem.data.cpu() for oelem in olist]
+    for i in range(len(olist) // 2):
+        ocls, oreg = olist[i * 2], olist[i * 2 + 1]
+        FB, FC, FH, FW = ocls.size()  # feature map size
+        stride = 2**(i + 2)    # 4,8,16,32,64,128
+        anchor = stride * 4
+        poss = zip(*np.where(ocls[:, 1, :, :] > 0.05))
+        for Iindex, hindex, windex in poss:
+            axc, ayc = stride / 2 + windex * stride, stride / 2 + hindex * stride
+            score = ocls[:, 1, hindex, windex]
+            loc = oreg[:, :, hindex, windex].contiguous().view(BB, 1, 4)
+            priors = torch.Tensor([[axc / 1.0, ayc / 1.0, stride * 4 / 1.0, stride * 4 / 1.0]]).view(1, 1, 4)
+            variances = [0.1, 0.2]
+            box = batch_decode(loc, priors, variances)
+            box = box[:, 0] * 1.0
+            # cv2.rectangle(imgshow,(int(x1),int(y1)),(int(x2),int(y2)),(0,0,255),1)
+            bboxlist.append(torch.cat([box, score.unsqueeze(1)], 1).cpu().numpy())
+    bboxlist = np.array(bboxlist)
+    if 0 == len(bboxlist):
+        bboxlist = np.zeros((1, BB, 5))
+
+    return bboxlist
+
+def flip_detect(net, img, device):
+    img = cv2.flip(img, 1)
+    b = detect(net, img, device)
+
+    bboxlist = np.zeros(b.shape)
+    bboxlist[:, 0] = img.shape[1] - b[:, 2]
+    bboxlist[:, 1] = b[:, 1]
+    bboxlist[:, 2] = img.shape[1] - b[:, 0]
+    bboxlist[:, 3] = b[:, 3]
+    bboxlist[:, 4] = b[:, 4]
+    return bboxlist
+
+
+def pts_to_bb(pts):
+    min_x, min_y = np.min(pts, axis=0)
+    max_x, max_y = np.max(pts, axis=0)
+    return np.array([min_x, min_y, max_x, max_y])

wav2lip/face_detection/detection/sfd/net_s3fd.py

@@ -0,0 +1,129 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class L2Norm(nn.Module):
+    def __init__(self, n_channels, scale=1.0):
+        super(L2Norm, self).__init__()
+        self.n_channels = n_channels
+        self.scale = scale
+        self.eps = 1e-10
+        self.weight = nn.Parameter(torch.Tensor(self.n_channels))
+        self.weight.data *= 0.0
+        self.weight.data += self.scale
+
+    def forward(self, x):
+        norm = x.pow(2).sum(dim=1, keepdim=True).sqrt() + self.eps
+        x = x / norm * self.weight.view(1, -1, 1, 1)
+        return x
+
+
+class s3fd(nn.Module):
+    def __init__(self):
+        super(s3fd, self).__init__()
+        self.conv1_1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
+        self.conv1_2 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
+
+        self.conv2_1 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
+        self.conv2_2 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1)
+
+        self.conv3_1 = nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1)
+        self.conv3_2 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)
+        self.conv3_3 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)
+
+        self.conv4_1 = nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1)
+        self.conv4_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
+        self.conv4_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
+
+        self.conv5_1 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
+        self.conv5_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
+        self.conv5_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
+
+        self.fc6 = nn.Conv2d(512, 1024, kernel_size=3, stride=1, padding=3)
+        self.fc7 = nn.Conv2d(1024, 1024, kernel_size=1, stride=1, padding=0)
+
+        self.conv6_1 = nn.Conv2d(1024, 256, kernel_size=1, stride=1, padding=0)
+        self.conv6_2 = nn.Conv2d(256, 512, kernel_size=3, stride=2, padding=1)
+
+        self.conv7_1 = nn.Conv2d(512, 128, kernel_size=1, stride=1, padding=0)
+        self.conv7_2 = nn.Conv2d(128, 256, kernel_size=3, stride=2, padding=1)
+
+        self.conv3_3_norm = L2Norm(256, scale=10)
+        self.conv4_3_norm = L2Norm(512, scale=8)
+        self.conv5_3_norm = L2Norm(512, scale=5)
+
+        self.conv3_3_norm_mbox_conf = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1)
+        self.conv3_3_norm_mbox_loc = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1)
+        self.conv4_3_norm_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1)
+        self.conv4_3_norm_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1)
+        self.conv5_3_norm_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1)
+        self.conv5_3_norm_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1)
+
+        self.fc7_mbox_conf = nn.Conv2d(1024, 2, kernel_size=3, stride=1, padding=1)
+        self.fc7_mbox_loc = nn.Conv2d(1024, 4, kernel_size=3, stride=1, padding=1)
+        self.conv6_2_mbox_conf = nn.Conv2d(512, 2, kernel_size=3, stride=1, padding=1)
+        self.conv6_2_mbox_loc = nn.Conv2d(512, 4, kernel_size=3, stride=1, padding=1)
+        self.conv7_2_mbox_conf = nn.Conv2d(256, 2, kernel_size=3, stride=1, padding=1)
+        self.conv7_2_mbox_loc = nn.Conv2d(256, 4, kernel_size=3, stride=1, padding=1)
+
+    def forward(self, x):
+        h = F.relu(self.conv1_1(x))
+        h = F.relu(self.conv1_2(h))
+        h = F.max_pool2d(h, 2, 2)
+
+        h = F.relu(self.conv2_1(h))
+        h = F.relu(self.conv2_2(h))
+        h = F.max_pool2d(h, 2, 2)
+
+        h = F.relu(self.conv3_1(h))
+        h = F.relu(self.conv3_2(h))
+        h = F.relu(self.conv3_3(h))
+        f3_3 = h
+        h = F.max_pool2d(h, 2, 2)
+
+        h = F.relu(self.conv4_1(h))
+        h = F.relu(self.conv4_2(h))
+        h = F.relu(self.conv4_3(h))
+        f4_3 = h
+        h = F.max_pool2d(h, 2, 2)
+
+        h = F.relu(self.conv5_1(h))
+        h = F.relu(self.conv5_2(h))
+        h = F.relu(self.conv5_3(h))
+        f5_3 = h
+        h = F.max_pool2d(h, 2, 2)
+
+        h = F.relu(self.fc6(h))
+        h = F.relu(self.fc7(h))
+        ffc7 = h
+        h = F.relu(self.conv6_1(h))
+        h = F.relu(self.conv6_2(h))
+        f6_2 = h
+        h = F.relu(self.conv7_1(h))
+        h = F.relu(self.conv7_2(h))
+        f7_2 = h
+
+        f3_3 = self.conv3_3_norm(f3_3)
+        f4_3 = self.conv4_3_norm(f4_3)
+        f5_3 = self.conv5_3_norm(f5_3)
+
+        cls1 = self.conv3_3_norm_mbox_conf(f3_3)
+        reg1 = self.conv3_3_norm_mbox_loc(f3_3)
+        cls2 = self.conv4_3_norm_mbox_conf(f4_3)
+        reg2 = self.conv4_3_norm_mbox_loc(f4_3)
+        cls3 = self.conv5_3_norm_mbox_conf(f5_3)
+        reg3 = self.conv5_3_norm_mbox_loc(f5_3)
+        cls4 = self.fc7_mbox_conf(ffc7)
+        reg4 = self.fc7_mbox_loc(ffc7)
+        cls5 = self.conv6_2_mbox_conf(f6_2)
+        reg5 = self.conv6_2_mbox_loc(f6_2)
+        cls6 = self.conv7_2_mbox_conf(f7_2)
+        reg6 = self.conv7_2_mbox_loc(f7_2)
+
+        # max-out background label
+        chunk = torch.chunk(cls1, 4, 1)
+        bmax = torch.max(torch.max(chunk[0], chunk[1]), chunk[2])
+        cls1 = torch.cat([bmax, chunk[3]], dim=1)
+
+        return [cls1, reg1, cls2, reg2, cls3, reg3, cls4, reg4, cls5, reg5, cls6, reg6]

wav2lip/face_detection/detection/sfd/sfd_detector.py

@@ -0,0 +1,72 @@
+import os
+import cv2
+from torch.utils.model_zoo import load_url
+
+from ..core import FaceDetector
+
+from .net_s3fd import s3fd
+from .bbox import *
+from .detect import *
+
+models_urls = {
+    's3fd': 'https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth',
+}
+
+
+class SFDDetector(FaceDetector):
+    def __init__(self, device, path_to_detector=os.path.join(os.path.dirname(os.path.abspath(__file__)), 's3fd.pth'), verbose=False):
+        super(SFDDetector, self).__init__(device, verbose)
+
+        # Initialise the face detector
+        try:
+            if not os.path.isfile(path_to_detector):
+                model_weights = load_url(models_urls['s3fd'])
+            else:
+                # For MPS (Apple Silicon), we need to load to CPU first
+                if 'mps' in device:
+                    model_weights = torch.load(path_to_detector, map_location='cpu')
+                else:
+                    model_weights = torch.load(path_to_detector, map_location=device)
+
+            self.face_detector = s3fd()
+            self.face_detector.load_state_dict(model_weights)
+            self.face_detector.to(device)
+            self.face_detector.eval()
+            
+            if verbose:
+                print(f"Face detector loaded successfully and moved to {device}")
+                
+        except Exception as e:
+            if verbose:
+                print(f"Error loading face detector model: {str(e)}")
+            raise
+
+    def detect_from_image(self, tensor_or_path):
+        image = self.tensor_or_path_to_ndarray(tensor_or_path)
+
+        bboxlist = detect(self.face_detector, image, device=self.device)
+        keep = nms(bboxlist, 0.3)
+        bboxlist = bboxlist[keep, :]
+        bboxlist = [x for x in bboxlist if x[-1] > 0.5]
+
+        return bboxlist
+
+    def detect_from_batch(self, images):
+        bboxlists = batch_detect(self.face_detector, images, device=self.device)
+        keeps = [nms(bboxlists[:, i, :], 0.3) for i in range(bboxlists.shape[1])]
+        bboxlists = [bboxlists[keep, i, :] for i, keep in enumerate(keeps)]
+        bboxlists = [[x for x in bboxlist if x[-1] > 0.5] for bboxlist in bboxlists]
+
+        return bboxlists
+
+    @property
+    def reference_scale(self):
+        return 195
+
+    @property
+    def reference_x_shift(self):
+        return 0
+
+    @property
+    def reference_y_shift(self):
+        return 0

wav2lip/face_detection/models.py

@@ -0,0 +1,261 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import math
+
+
+def conv3x3(in_planes, out_planes, strd=1, padding=1, bias=False):
+    "3x3 convolution with padding"
+    return nn.Conv2d(in_planes, out_planes, kernel_size=3,
+                     stride=strd, padding=padding, bias=bias)
+
+
+class ConvBlock(nn.Module):
+    def __init__(self, in_planes, out_planes):
+        super(ConvBlock, self).__init__()
+        self.bn1 = nn.BatchNorm2d(in_planes)
+        self.conv1 = conv3x3(in_planes, int(out_planes / 2))
+        self.bn2 = nn.BatchNorm2d(int(out_planes / 2))
+        self.conv2 = conv3x3(int(out_planes / 2), int(out_planes / 4))
+        self.bn3 = nn.BatchNorm2d(int(out_planes / 4))
+        self.conv3 = conv3x3(int(out_planes / 4), int(out_planes / 4))
+
+        if in_planes != out_planes:
+            self.downsample = nn.Sequential(
+                nn.BatchNorm2d(in_planes),
+                nn.ReLU(True),
+                nn.Conv2d(in_planes, out_planes,
+                          kernel_size=1, stride=1, bias=False),
+            )
+        else:
+            self.downsample = None
+
+    def forward(self, x):
+        residual = x
+
+        out1 = self.bn1(x)
+        out1 = F.relu(out1, True)
+        out1 = self.conv1(out1)
+
+        out2 = self.bn2(out1)
+        out2 = F.relu(out2, True)
+        out2 = self.conv2(out2)
+
+        out3 = self.bn3(out2)
+        out3 = F.relu(out3, True)
+        out3 = self.conv3(out3)
+
+        out3 = torch.cat((out1, out2, out3), 1)
+
+        if self.downsample is not None:
+            residual = self.downsample(residual)
+
+        out3 += residual
+
+        return out3
+
+
+class Bottleneck(nn.Module):
+
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1, downsample=None):
+        super(Bottleneck, self).__init__()
+        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
+                               padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * 4)
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = downsample
+        self.stride = stride
+
+    def forward(self, x):
+        residual = x
+
+        out = self.conv1(x)
+        out = self.bn1(out)
+        out = self.relu(out)
+
+        out = self.conv2(out)
+        out = self.bn2(out)
+        out = self.relu(out)
+
+        out = self.conv3(out)
+        out = self.bn3(out)
+
+        if self.downsample is not None:
+            residual = self.downsample(x)
+
+        out += residual
+        out = self.relu(out)
+
+        return out
+
+
+class HourGlass(nn.Module):
+    def __init__(self, num_modules, depth, num_features):
+        super(HourGlass, self).__init__()
+        self.num_modules = num_modules
+        self.depth = depth
+        self.features = num_features
+
+        self._generate_network(self.depth)
+
+    def _generate_network(self, level):
+        self.add_module('b1_' + str(level), ConvBlock(self.features, self.features))
+
+        self.add_module('b2_' + str(level), ConvBlock(self.features, self.features))
+
+        if level > 1:
+            self._generate_network(level - 1)
+        else:
+            self.add_module('b2_plus_' + str(level), ConvBlock(self.features, self.features))
+
+        self.add_module('b3_' + str(level), ConvBlock(self.features, self.features))
+
+    def _forward(self, level, inp):
+        # Upper branch
+        up1 = inp
+        up1 = self._modules['b1_' + str(level)](up1)
+
+        # Lower branch
+        low1 = F.avg_pool2d(inp, 2, stride=2)
+        low1 = self._modules['b2_' + str(level)](low1)
+
+        if level > 1:
+            low2 = self._forward(level - 1, low1)
+        else:
+            low2 = low1
+            low2 = self._modules['b2_plus_' + str(level)](low2)
+
+        low3 = low2
+        low3 = self._modules['b3_' + str(level)](low3)
+
+        up2 = F.interpolate(low3, scale_factor=2, mode='nearest')
+
+        return up1 + up2
+
+    def forward(self, x):
+        return self._forward(self.depth, x)
+
+
+class FAN(nn.Module):
+
+    def __init__(self, num_modules=1):
+        super(FAN, self).__init__()
+        self.num_modules = num_modules
+
+        # Base part
+        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)
+        self.bn1 = nn.BatchNorm2d(64)
+        self.conv2 = ConvBlock(64, 128)
+        self.conv3 = ConvBlock(128, 128)
+        self.conv4 = ConvBlock(128, 256)
+
+        # Stacking part
+        for hg_module in range(self.num_modules):
+            self.add_module('m' + str(hg_module), HourGlass(1, 4, 256))
+            self.add_module('top_m_' + str(hg_module), ConvBlock(256, 256))
+            self.add_module('conv_last' + str(hg_module),
+                            nn.Conv2d(256, 256, kernel_size=1, stride=1, padding=0))
+            self.add_module('bn_end' + str(hg_module), nn.BatchNorm2d(256))
+            self.add_module('l' + str(hg_module), nn.Conv2d(256,
+                                                            68, kernel_size=1, stride=1, padding=0))
+
+            if hg_module < self.num_modules - 1:
+                self.add_module(
+                    'bl' + str(hg_module), nn.Conv2d(256, 256, kernel_size=1, stride=1, padding=0))
+                self.add_module('al' + str(hg_module), nn.Conv2d(68,
+                                                                 256, kernel_size=1, stride=1, padding=0))
+
+    def forward(self, x):
+        x = F.relu(self.bn1(self.conv1(x)), True)
+        x = F.avg_pool2d(self.conv2(x), 2, stride=2)
+        x = self.conv3(x)
+        x = self.conv4(x)
+
+        previous = x
+
+        outputs = []
+        for i in range(self.num_modules):
+            hg = self._modules['m' + str(i)](previous)
+
+            ll = hg
+            ll = self._modules['top_m_' + str(i)](ll)
+
+            ll = F.relu(self._modules['bn_end' + str(i)]
+                        (self._modules['conv_last' + str(i)](ll)), True)
+
+            # Predict heatmaps
+            tmp_out = self._modules['l' + str(i)](ll)
+            outputs.append(tmp_out)
+
+            if i < self.num_modules - 1:
+                ll = self._modules['bl' + str(i)](ll)
+                tmp_out_ = self._modules['al' + str(i)](tmp_out)
+                previous = previous + ll + tmp_out_
+
+        return outputs
+
+
+class ResNetDepth(nn.Module):
+
+    def __init__(self, block=Bottleneck, layers=[3, 8, 36, 3], num_classes=68):
+        self.inplanes = 64
+        super(ResNetDepth, self).__init__()
+        self.conv1 = nn.Conv2d(3 + 68, 64, kernel_size=7, stride=2, padding=3,
+                               bias=False)
+        self.bn1 = nn.BatchNorm2d(64)
+        self.relu = nn.ReLU(inplace=True)
+        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
+        self.layer1 = self._make_layer(block, 64, layers[0])
+        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
+        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
+        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
+        self.avgpool = nn.AvgPool2d(7)
+        self.fc = nn.Linear(512 * block.expansion, num_classes)
+
+        for m in self.modules():
+            if isinstance(m, nn.Conv2d):
+                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
+                m.weight.data.normal_(0, math.sqrt(2. / n))
+            elif isinstance(m, nn.BatchNorm2d):
+                m.weight.data.fill_(1)
+                m.bias.data.zero_()
+
+    def _make_layer(self, block, planes, blocks, stride=1):
+        downsample = None
+        if stride != 1 or self.inplanes != planes * block.expansion:
+            downsample = nn.Sequential(
+                nn.Conv2d(self.inplanes, planes * block.expansion,
+                          kernel_size=1, stride=stride, bias=False),
+                nn.BatchNorm2d(planes * block.expansion),
+            )
+
+        layers = []
+        layers.append(block(self.inplanes, planes, stride, downsample))
+        self.inplanes = planes * block.expansion
+        for i in range(1, blocks):
+            layers.append(block(self.inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = self.bn1(x)
+        x = self.relu(x)
+        x = self.maxpool(x)
+
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+        x = self.fc(x)
+
+        return x

wav2lip/face_detection/utils.py

@@ -0,0 +1,313 @@
+from __future__ import print_function
+import os
+import sys
+import time
+import torch
+import math
+import numpy as np
+import cv2
+
+
+def _gaussian(
+        size=3, sigma=0.25, amplitude=1, normalize=False, width=None,
+        height=None, sigma_horz=None, sigma_vert=None, mean_horz=0.5,
+        mean_vert=0.5):
+    # handle some defaults
+    if width is None:
+        width = size
+    if height is None:
+        height = size
+    if sigma_horz is None:
+        sigma_horz = sigma
+    if sigma_vert is None:
+        sigma_vert = sigma
+    center_x = mean_horz * width + 0.5
+    center_y = mean_vert * height + 0.5
+    gauss = np.empty((height, width), dtype=np.float32)
+    # generate kernel
+    for i in range(height):
+        for j in range(width):
+            gauss[i][j] = amplitude * math.exp(-(math.pow((j + 1 - center_x) / (
+                sigma_horz * width), 2) / 2.0 + math.pow((i + 1 - center_y) / (sigma_vert * height), 2) / 2.0))
+    if normalize:
+        gauss = gauss / np.sum(gauss)
+    return gauss
+
+
+def draw_gaussian(image, point, sigma):
+    # Check if the gaussian is inside
+    ul = [math.floor(point[0] - 3 * sigma), math.floor(point[1] - 3 * sigma)]
+    br = [math.floor(point[0] + 3 * sigma), math.floor(point[1] + 3 * sigma)]
+    if (ul[0] > image.shape[1] or ul[1] > image.shape[0] or br[0] < 1 or br[1] < 1):
+        return image
+    size = 6 * sigma + 1
+    g = _gaussian(size)
+    g_x = [int(max(1, -ul[0])), int(min(br[0], image.shape[1])) - int(max(1, ul[0])) + int(max(1, -ul[0]))]
+    g_y = [int(max(1, -ul[1])), int(min(br[1], image.shape[0])) - int(max(1, ul[1])) + int(max(1, -ul[1]))]
+    img_x = [int(max(1, ul[0])), int(min(br[0], image.shape[1]))]
+    img_y = [int(max(1, ul[1])), int(min(br[1], image.shape[0]))]
+    assert (g_x[0] > 0 and g_y[1] > 0)
+    image[img_y[0] - 1:img_y[1], img_x[0] - 1:img_x[1]
+          ] = image[img_y[0] - 1:img_y[1], img_x[0] - 1:img_x[1]] + g[g_y[0] - 1:g_y[1], g_x[0] - 1:g_x[1]]
+    image[image > 1] = 1
+    return image
+
+
+def transform(point, center, scale, resolution, invert=False):
+    """Generate and affine transformation matrix.
+
+    Given a set of points, a center, a scale and a targer resolution, the
+    function generates and affine transformation matrix. If invert is ``True``
+    it will produce the inverse transformation.
+
+    Arguments:
+        point {torch.tensor} -- the input 2D point
+        center {torch.tensor or numpy.array} -- the center around which to perform the transformations
+        scale {float} -- the scale of the face/object
+        resolution {float} -- the output resolution
+
+    Keyword Arguments:
+        invert {bool} -- define wherever the function should produce the direct or the
+        inverse transformation matrix (default: {False})
+    """
+    _pt = torch.ones(3)
+    _pt[0] = point[0]
+    _pt[1] = point[1]
+
+    h = 200.0 * scale
+    t = torch.eye(3)
+    t[0, 0] = resolution / h
+    t[1, 1] = resolution / h
+    t[0, 2] = resolution * (-center[0] / h + 0.5)
+    t[1, 2] = resolution * (-center[1] / h + 0.5)
+
+    if invert:
+        t = torch.inverse(t)
+
+    new_point = (torch.matmul(t, _pt))[0:2]
+
+    return new_point.int()
+
+
+def crop(image, center, scale, resolution=256.0):
+    """Center crops an image or set of heatmaps
+
+    Arguments:
+        image {numpy.array} -- an rgb image
+        center {numpy.array} -- the center of the object, usually the same as of the bounding box
+        scale {float} -- scale of the face
+
+    Keyword Arguments:
+        resolution {float} -- the size of the output cropped image (default: {256.0})
+
+    Returns:
+        [type] -- [description]
+    """  # Crop around the center point
+    """ Crops the image around the center. Input is expected to be an np.ndarray """
+    ul = transform([1, 1], center, scale, resolution, True)
+    br = transform([resolution, resolution], center, scale, resolution, True)
+    # pad = math.ceil(torch.norm((ul - br).float()) / 2.0 - (br[0] - ul[0]) / 2.0)
+    if image.ndim > 2:
+        newDim = np.array([br[1] - ul[1], br[0] - ul[0],
+                           image.shape[2]], dtype=np.int32)
+        newImg = np.zeros(newDim, dtype=np.uint8)
+    else:
+        newDim = np.array([br[1] - ul[1], br[0] - ul[0]], dtype=np.int)
+        newImg = np.zeros(newDim, dtype=np.uint8)
+    ht = image.shape[0]
+    wd = image.shape[1]
+    newX = np.array(
+        [max(1, -ul[0] + 1), min(br[0], wd) - ul[0]], dtype=np.int32)
+    newY = np.array(
+        [max(1, -ul[1] + 1), min(br[1], ht) - ul[1]], dtype=np.int32)
+    oldX = np.array([max(1, ul[0] + 1), min(br[0], wd)], dtype=np.int32)
+    oldY = np.array([max(1, ul[1] + 1), min(br[1], ht)], dtype=np.int32)
+    newImg[newY[0] - 1:newY[1], newX[0] - 1:newX[1]
+           ] = image[oldY[0] - 1:oldY[1], oldX[0] - 1:oldX[1], :]
+    newImg = cv2.resize(newImg, dsize=(int(resolution), int(resolution)),
+                        interpolation=cv2.INTER_LINEAR)
+    return newImg
+
+
+def get_preds_fromhm(hm, center=None, scale=None):
+    """Obtain (x,y) coordinates given a set of N heatmaps. If the center
+    and the scale is provided the function will return the points also in
+    the original coordinate frame.
+
+    Arguments:
+        hm {torch.tensor} -- the predicted heatmaps, of shape [B, N, W, H]
+
+    Keyword Arguments:
+        center {torch.tensor} -- the center of the bounding box (default: {None})
+        scale {float} -- face scale (default: {None})
+    """
+    max, idx = torch.max(
+        hm.view(hm.size(0), hm.size(1), hm.size(2) * hm.size(3)), 2)
+    idx += 1
+    preds = idx.view(idx.size(0), idx.size(1), 1).repeat(1, 1, 2).float()
+    preds[..., 0].apply_(lambda x: (x - 1) % hm.size(3) + 1)
+    preds[..., 1].add_(-1).div_(hm.size(2)).floor_().add_(1)
+
+    for i in range(preds.size(0)):
+        for j in range(preds.size(1)):
+            hm_ = hm[i, j, :]
+            pX, pY = int(preds[i, j, 0]) - 1, int(preds[i, j, 1]) - 1
+            if pX > 0 and pX < 63 and pY > 0 and pY < 63:
+                diff = torch.FloatTensor(
+                    [hm_[pY, pX + 1] - hm_[pY, pX - 1],
+                     hm_[pY + 1, pX] - hm_[pY - 1, pX]])
+                preds[i, j].add_(diff.sign_().mul_(.25))
+
+    preds.add_(-.5)
+
+    preds_orig = torch.zeros(preds.size())
+    if center is not None and scale is not None:
+        for i in range(hm.size(0)):
+            for j in range(hm.size(1)):
+                preds_orig[i, j] = transform(
+                    preds[i, j], center, scale, hm.size(2), True)
+
+    return preds, preds_orig
+
+def get_preds_fromhm_batch(hm, centers=None, scales=None):
+    """Obtain (x,y) coordinates given a set of N heatmaps. If the centers
+    and the scales is provided the function will return the points also in
+    the original coordinate frame.
+
+    Arguments:
+        hm {torch.tensor} -- the predicted heatmaps, of shape [B, N, W, H]
+
+    Keyword Arguments:
+        centers {torch.tensor} -- the centers of the bounding box (default: {None})
+        scales {float} -- face scales (default: {None})
+    """
+    max, idx = torch.max(
+        hm.view(hm.size(0), hm.size(1), hm.size(2) * hm.size(3)), 2)
+    idx += 1
+    preds = idx.view(idx.size(0), idx.size(1), 1).repeat(1, 1, 2).float()
+    preds[..., 0].apply_(lambda x: (x - 1) % hm.size(3) + 1)
+    preds[..., 1].add_(-1).div_(hm.size(2)).floor_().add_(1)
+
+    for i in range(preds.size(0)):
+        for j in range(preds.size(1)):
+            hm_ = hm[i, j, :]
+            pX, pY = int(preds[i, j, 0]) - 1, int(preds[i, j, 1]) - 1
+            if pX > 0 and pX < 63 and pY > 0 and pY < 63:
+                diff = torch.FloatTensor(
+                    [hm_[pY, pX + 1] - hm_[pY, pX - 1],
+                     hm_[pY + 1, pX] - hm_[pY - 1, pX]])
+                preds[i, j].add_(diff.sign_().mul_(.25))
+
+    preds.add_(-.5)
+
+    preds_orig = torch.zeros(preds.size())
+    if centers is not None and scales is not None:
+        for i in range(hm.size(0)):
+            for j in range(hm.size(1)):
+                preds_orig[i, j] = transform(
+                    preds[i, j], centers[i], scales[i], hm.size(2), True)
+
+    return preds, preds_orig
+
+def shuffle_lr(parts, pairs=None):
+    """Shuffle the points left-right according to the axis of symmetry
+    of the object.
+
+    Arguments:
+        parts {torch.tensor} -- a 3D or 4D object containing the
+        heatmaps.
+
+    Keyword Arguments:
+        pairs {list of integers} -- [order of the flipped points] (default: {None})
+    """
+    if pairs is None:
+        pairs = [16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0,
+                 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 27, 28, 29, 30, 35,
+                 34, 33, 32, 31, 45, 44, 43, 42, 47, 46, 39, 38, 37, 36, 41,
+                 40, 54, 53, 52, 51, 50, 49, 48, 59, 58, 57, 56, 55, 64, 63,
+                 62, 61, 60, 67, 66, 65]
+    if parts.ndimension() == 3:
+        parts = parts[pairs, ...]
+    else:
+        parts = parts[:, pairs, ...]
+
+    return parts
+
+
+def flip(tensor, is_label=False):
+    """Flip an image or a set of heatmaps left-right
+
+    Arguments:
+        tensor {numpy.array or torch.tensor} -- [the input image or heatmaps]
+
+    Keyword Arguments:
+        is_label {bool} -- [denote wherever the input is an image or a set of heatmaps ] (default: {False})
+    """
+    if not torch.is_tensor(tensor):
+        tensor = torch.from_numpy(tensor)
+
+    if is_label:
+        tensor = shuffle_lr(tensor).flip(tensor.ndimension() - 1)
+    else:
+        tensor = tensor.flip(tensor.ndimension() - 1)
+
+    return tensor
+
+# From pyzolib/paths.py (https://bitbucket.org/pyzo/pyzolib/src/tip/paths.py)
+
+
+def appdata_dir(appname=None, roaming=False):
+    """ appdata_dir(appname=None, roaming=False)
+
+    Get the path to the application directory, where applications are allowed
+    to write user specific files (e.g. configurations). For non-user specific
+    data, consider using common_appdata_dir().
+    If appname is given, a subdir is appended (and created if necessary).
+    If roaming is True, will prefer a roaming directory (Windows Vista/7).
+    """
+
+    # Define default user directory
+    userDir = os.getenv('FACEALIGNMENT_USERDIR', None)
+    if userDir is None:
+        userDir = os.path.expanduser('~')
+        if not os.path.isdir(userDir):  # pragma: no cover
+            userDir = '/var/tmp'  # issue #54
+
+    # Get system app data dir
+    path = None
+    if sys.platform.startswith('win'):
+        path1, path2 = os.getenv('LOCALAPPDATA'), os.getenv('APPDATA')
+        path = (path2 or path1) if roaming else (path1 or path2)
+    elif sys.platform.startswith('darwin'):
+        path = os.path.join(userDir, 'Library', 'Application Support')
+    # On Linux and as fallback
+    if not (path and os.path.isdir(path)):
+        path = userDir
+
+    # Maybe we should store things local to the executable (in case of a
+    # portable distro or a frozen application that wants to be portable)
+    prefix = sys.prefix
+    if getattr(sys, 'frozen', None):
+        prefix = os.path.abspath(os.path.dirname(sys.executable))
+    for reldir in ('settings', '../settings'):
+        localpath = os.path.abspath(os.path.join(prefix, reldir))
+        if os.path.isdir(localpath):  # pragma: no cover
+            try:
+                open(os.path.join(localpath, 'test.write'), 'wb').close()
+                os.remove(os.path.join(localpath, 'test.write'))
+            except IOError:
+                pass  # We cannot write in this directory
+            else:
+                path = localpath
+                break
+
+    # Get path specific for this app
+    if appname:
+        if path == userDir:
+            appname = '.' + appname.lstrip('.')  # Make it a hidden directory
+        path = os.path.join(path, appname)
+        if not os.path.isdir(path):  # pragma: no cover
+            os.mkdir(path)
+
+    # Done
+    return path

wav2lip/hparams.py

@@ -0,0 +1,101 @@
+from glob import glob
+import os
+
+def get_image_list(data_root, split):
+	filelist = []
+
+	with open('filelists/{}.txt'.format(split)) as f:
+		for line in f:
+			line = line.strip()
+			if ' ' in line: line = line.split()[0]
+			filelist.append(os.path.join(data_root, line))
+
+	return filelist
+
+class HParams:
+	def __init__(self, **kwargs):
+		self.data = {}
+
+		for key, value in kwargs.items():
+			self.data[key] = value
+
+	def __getattr__(self, key):
+		if key not in self.data:
+			raise AttributeError("'HParams' object has no attribute %s" % key)
+		return self.data[key]
+
+	def set_hparam(self, key, value):
+		self.data[key] = value
+
+
+# Default hyperparameters
+hparams = HParams(
+	num_mels=80,  # Number of mel-spectrogram channels and local conditioning dimensionality
+	#  network
+	rescale=True,  # Whether to rescale audio prior to preprocessing
+	rescaling_max=0.9,  # Rescaling value
+	
+	# Use LWS (https://github.com/Jonathan-LeRoux/lws) for STFT and phase reconstruction
+	# It"s preferred to set True to use with https://github.com/r9y9/wavenet_vocoder
+	# Does not work if n_ffit is not multiple of hop_size!!
+	use_lws=False,
+	
+	n_fft=800,  # Extra window size is filled with 0 paddings to match this parameter
+	hop_size=200,  # For 16000Hz, 200 = 12.5 ms (0.0125 * sample_rate)
+	win_size=800,  # For 16000Hz, 800 = 50 ms (If None, win_size = n_fft) (0.05 * sample_rate)
+	sample_rate=16000,  # 16000Hz (corresponding to librispeech) (sox --i <filename>)
+	
+	frame_shift_ms=None,  # Can replace hop_size parameter. (Recommended: 12.5)
+	
+	# Mel and Linear spectrograms normalization/scaling and clipping
+	signal_normalization=True,
+	# Whether to normalize mel spectrograms to some predefined range (following below parameters)
+	allow_clipping_in_normalization=True,  # Only relevant if mel_normalization = True
+	symmetric_mels=True,
+	# Whether to scale the data to be symmetric around 0. (Also multiplies the output range by 2, 
+	# faster and cleaner convergence)
+	max_abs_value=4.,
+	# max absolute value of data. If symmetric, data will be [-max, max] else [0, max] (Must not 
+	# be too big to avoid gradient explosion, 
+	# not too small for fast convergence)
+	# Contribution by @begeekmyfriend
+	# Spectrogram Pre-Emphasis (Lfilter: Reduce spectrogram noise and helps model certitude 
+	# levels. Also allows for better G&L phase reconstruction)
+	preemphasize=True,  # whether to apply filter
+	preemphasis=0.97,  # filter coefficient.
+	
+	# Limits
+	min_level_db=-100,
+	ref_level_db=20,
+	fmin=55,
+	# Set this to 55 if your speaker is male! if female, 95 should help taking off noise. (To 
+	# test depending on dataset. Pitch info: male~[65, 260], female~[100, 525])
+	fmax=7600,  # To be increased/reduced depending on data.
+
+	###################### Our training parameters #################################
+	img_size=96,
+	fps=25,
+	
+	batch_size=16,
+	initial_learning_rate=1e-4,
+	nepochs=200000000000000000,  ### ctrl + c, stop whenever eval loss is consistently greater than train loss for ~10 epochs
+	num_workers=16,
+	checkpoint_interval=3000,
+	eval_interval=3000,
+    save_optimizer_state=True,
+
+    syncnet_wt=0.0, # is initially zero, will be set automatically to 0.03 later. Leads to faster convergence. 
+	syncnet_batch_size=64,
+	syncnet_lr=1e-4,
+	syncnet_eval_interval=10000,
+	syncnet_checkpoint_interval=10000,
+
+	disc_wt=0.07,
+	disc_initial_learning_rate=1e-4,
+)
+
+
+def hparams_debug_string():
+	values = hparams.values()
+	hp = ["  %s: %s" % (name, values[name]) for name in sorted(values) if name != "sentences"]
+	return "Hyperparameters:\n" + "\n".join(hp)

wav2lip/inference.py

@@ -0,0 +1,484 @@
+import numpy as np
+import cv2 
+import os 
+import argparse
+import subprocess
+from tqdm import tqdm
+import sys
+import traceback
+from .audio import load_wav, melspectrogram
+from .face_detection import FaceAlignment, LandmarksType
+import torch
+import platform
+
+parser = argparse.ArgumentParser(description='Inference code to lip-sync videos in the wild using Wav2Lip models')
+
+parser.add_argument('--outfile', type=str, help='Video path to save result. See default for an e.g.', 
+								default='wav2lip/results/result_voice.mp4')
+
+parser.add_argument('--static', type=bool, 
+					help='If True, then use only first video frame for inference', default=False)
+parser.add_argument('--fps', type=float, help='Can be specified only if input is a static image (default: 25)', 
+					default=25., required=False)
+
+parser.add_argument('--pads', nargs='+', type=int, default=[0, 10, 0, 0], 
+					help='Padding (top, bottom, left, right). Please adjust to include chin at least')
+
+parser.add_argument('--face_det_batch_size', type=int, 
+					help='Batch size for face detection', default=32)
+parser.add_argument('--wav2lip_batch_size', type=int, help='Batch size for Wav2Lip model(s)', default=512)
+
+parser.add_argument('--resize_factor', default=1, type=int, 
+			help='Reduce the resolution by this factor. Sometimes, best results are obtained at 480p or 720p')
+
+parser.add_argument('--crop', nargs='+', type=int, default=[0, -1, 0, -1], 
+					help='Crop video to a smaller region (top, bottom, left, right). Applied after resize_factor and rotate arg. ' 
+					'Useful if multiple face present. -1 implies the value will be auto-inferred based on height, width')
+
+parser.add_argument('--box', nargs='+', type=int, default=[-1, -1, -1, -1], 
+					help='Specify a constant bounding box for the face. Use only as a last resort if the face is not detected.'
+					'Also, might work only if the face is not moving around much. Syntax: (top, bottom, left, right).')
+
+parser.add_argument('--rotate', default=False, action='store_true',
+					help='Sometimes videos taken from a phone can be flipped 90deg. If true, will flip video right by 90deg.'
+					'Use if you get a flipped result, despite feeding a normal looking video')
+
+parser.add_argument('--nosmooth', default=False, action='store_true',
+					help='Prevent smoothing face detections over a short temporal window')
+
+args = parser.parse_args()
+args.img_size = 96
+
+# Check for available devices
+if torch.backends.mps.is_available():
+    device = 'mps'  # Use Apple Silicon GPU
+elif torch.cuda.is_available():
+    device = 'cuda'
+else:
+    device = 'cpu'
+
+print('Using {} for inference.'.format(device))
+
+def get_smoothened_boxes(boxes, idx):
+    """Get smoothened box for a specific index"""
+    if idx >= len(boxes) or boxes[idx] is None:
+        return None, None
+    
+    # Return the face region and coordinates
+    if isinstance(boxes[idx], list) and len(boxes[idx]) == 2:  # Format from the specified bounding box
+        return boxes[idx][0], boxes[idx][1]
+    else:  # Format from face detection - [x1, y1, x2, y2]
+        if isinstance(boxes[idx], list) or isinstance(boxes[idx], tuple):
+            if len(boxes[idx]) >= 4:  # Make sure we have all 4 coordinates
+                x1, y1, x2, y2 = boxes[idx][:4]
+                # Return coordinates in the expected format (y1, y2, x1, x2)
+                coords = (y1, y2, x1, x2)
+                return None, coords
+        
+        print(f"WARNING: Unexpected box format at idx {idx}: {boxes[idx]}")
+        return None, None
+
+def face_detect(images):
+    print(f"Starting face detection using {device} device...")
+    try:
+        detector = FaceAlignment(LandmarksType._2D, 
+                                flip_input=False, device=device, verbose=True)
+    except Exception as e:
+        print(f"Error initializing face detector: {str(e)}")
+        print("Attempting to fall back to CPU for face detection...")
+        detector = FaceAlignment(LandmarksType._2D, 
+                                flip_input=False, device='cpu', verbose=True)
+    
+    batch_size = args.face_det_batch_size
+    
+    while 1:
+        predictions = []
+        try:
+            for i in range(0, len(images), batch_size):
+                batch = np.array(images[i:i + batch_size])
+                print(f"Processing detection batch {i//batch_size + 1}, shape: {batch.shape}")
+                batch_predictions = detector.get_detections_for_batch(batch)
+                predictions.extend(batch_predictions)
+        except RuntimeError as e:
+            print(f"Runtime error in face detection: {str(e)}")
+            if batch_size == 1:
+                # Error when batch_size is already 1
+                print('Face detection failed at minimum batch size! Using fallback method...')
+                # Create empty predictions for all frames to allow processing to continue
+                predictions = [None] * len(images)
+                break
+            batch_size //= 2
+            print('Reducing face detection batch size to', batch_size)
+            continue
+        except Exception as e:
+            print(f"Unexpected error in face detection: {str(e)}")
+            # Create empty predictions and continue with fallback
+            predictions = [None] * len(images)
+            break
+        break
+        
+    # Check if we have at least one valid face detection
+    faces_detected = sum(1 for p in predictions if p is not None)
+    print(f"Detected faces in {faces_detected} out of {len(images)} frames ({faces_detected/len(images)*100:.1f}%)")
+    
+    results = []
+    pady1, pady2, padx1, padx2 = args.pads
+    
+    for i, (rect, image) in enumerate(zip(predictions, images)):
+        if rect is None:
+            # Create default coordinates for face detection
+            h, w = image.shape[:2]
+            
+            # Simple and consistent face region estimation based on center of the frame
+            center_x = w // 2
+            center_y = h // 2
+            
+            # Use about 1/3 of the frame height for face
+            face_h = h // 3
+            face_w = min(w // 2, face_h)
+            
+            # Create a centered box
+            x1 = max(0, center_x - face_w // 2 - padx1)
+            y1 = max(0, center_y - face_h // 2 - pady1)
+            x2 = min(w, center_x + face_w // 2 + padx2)
+            y2 = min(h, center_y + face_h // 2 + pady2)
+            
+            if i == 0 or i % 100 == 0:  # Log only occasionally to avoid flooding
+                print(f"Frame {i}: Using fallback face region at ({x1},{y1},{x2},{y2})")
+            
+            results.append([x1, y1, x2, y2])
+            continue
+            
+        # If face is detected, use its coordinates with padding
+        y1 = max(0, rect[1] - pady1)
+        y2 = min(image.shape[0], rect[3] + pady2)
+        x1 = max(0, rect[0] - padx1)
+        x2 = min(image.shape[1], rect[2] + padx2)
+        
+        results.append([x1, y1, x2, y2])
+    
+    return results
+
+def datagen(frames, mels):
+	img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []
+
+	if args.box[0] == -1:
+		if not args.static:
+			try:
+				print(f"Starting face detection for {len(frames)} frames...")
+				face_det_results = face_detect(frames) # BGR2RGB for CNN face detection
+				print("Face detection completed successfully")
+			except Exception as e:
+				print(f"Face detection error: {str(e)}")
+				print(f"Error type: {type(e).__name__}")
+				traceback.print_exc()
+				print("Using fallback method with default face regions...")
+				# Create default face regions for all frames
+				h, w = frames[0].shape[:2]
+				
+				# Simple face region estimation in the center of the frame
+				center_x = w // 2
+				center_y = h // 2
+				
+				# Use about 1/3 of the frame height for face
+				face_h = h // 3
+				face_w = min(w // 2, face_h)
+				
+				pady1, pady2, padx1, padx2 = args.pads
+				x1 = max(0, center_x - face_w // 2 - padx1)
+				y1 = max(0, center_y - face_h // 2 - pady1)
+				x2 = min(w, center_x + face_w // 2 + padx2)
+				y2 = min(h, center_y + face_h // 2 + pady2)
+				
+				print(f"Estimated face region: x1={x1}, y1={y1}, x2={x2}, y2={y2}")
+				
+				# Use the same format as the face_detect function returns
+				face_det_results = [[x1, y1, x2, y2] for _ in range(len(frames))]
+		else:
+			try:
+				print("Starting face detection for static image...")
+				face_det_results = face_detect([frames[0]])
+				print("Face detection completed successfully")
+			except Exception as e:
+				print(f"Face detection error: {str(e)}")
+				print(f"Error type: {type(e).__name__}")
+				traceback.print_exc()
+				print("Using fallback method with default face region...")
+				# Create default face region for static image
+				h, w = frames[0].shape[:2]
+				
+				# Simple face region estimation in the center of the frame
+				center_x = w // 2
+				center_y = h // 2
+				
+				# Use about 1/3 of the frame height for face
+				face_h = h // 3
+				face_w = min(w // 2, face_h)
+				
+				pady1, pady2, padx1, padx2 = args.pads
+				x1 = max(0, center_x - face_w // 2 - padx1)
+				y1 = max(0, center_y - face_h // 2 - pady1)
+				x2 = min(w, center_x + face_w // 2 + padx2)
+				y2 = min(h, center_y + face_h // 2 + pady2)
+				
+				print(f"Estimated face region for static image: x1={x1}, y1={y1}, x2={x2}, y2={y2}")
+				
+				# Use the same format as the face_detect function returns
+				face_det_results = [[x1, y1, x2, y2]]
+	else:
+		print('Using the specified bounding box instead of face detection...')
+		y1, y2, x1, x2 = args.box
+		face_det_results = [[x1, y1, x2, y2] for _ in range(len(frames))]
+
+	for i, m in enumerate(mels):
+		idx = 0 if args.static else i%len(frames)
+		frame_to_save = frames[idx].copy()
+		
+		if args.box[0] == -1:
+			face, coords = get_smoothened_boxes(face_det_results, idx)
+			
+			if coords is None:
+				print(f'Face coordinates not detected! Skipping frame {i}')
+				continue
+			
+			# If face is None, extract it from the frame using coordinates
+			if face is None:
+				y1, y2, x1, x2 = coords
+				try:
+					if y1 >= y2 or x1 >= x2:
+						print(f"Invalid coordinates at frame {i}: y1={y1}, y2={y2}, x1={x1}, x2={x2}")
+						continue
+					if y1 < 0 or x1 < 0 or y2 > frame_to_save.shape[0] or x2 > frame_to_save.shape[1]:
+						print(f"Out of bounds coordinates at frame {i}. Adjusting...")
+						y1 = max(0, y1)
+						x1 = max(0, x1)
+						y2 = min(frame_to_save.shape[0], y2)
+						x2 = min(frame_to_save.shape[1], x2)
+					
+					# Check if the region is too small
+					if (y2 - y1) < 10 or (x2 - x1) < 10:
+						print(f"Region too small at frame {i}. Skipping.")
+						continue
+						
+					face = frames[idx][y1:y2, x1:x2]
+				except Exception as e:
+					print(f"Error extracting face at frame {i}: {str(e)}")
+					continue
+		else:
+			face = frames[idx][y1:y2, x1:x2]
+			coords = (y1, y2, x1, x2)
+			
+		try:    
+			face = cv2.resize(face, (args.img_size, args.img_size))
+			img_batch.append(face)
+			mel_batch.append(m)
+			frame_batch.append(frame_to_save)
+			coords_batch.append(coords)
+		except Exception as e:
+			print(f"Error processing frame {i}: {str(e)}")
+			continue
+		
+		if len(img_batch) >= args.wav2lip_batch_size:
+			img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)
+
+			img_masked = img_batch.copy()
+			img_masked[:, args.img_size//2:] = 0
+
+			img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
+			mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])
+
+			yield img_batch, mel_batch, frame_batch, coords_batch
+			img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []
+
+	if len(img_batch) > 0:
+		img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)
+
+		img_masked = img_batch.copy()
+		img_masked[:, args.img_size//2:] = 0
+
+		img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
+		mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])
+
+		yield img_batch, mel_batch, frame_batch, coords_batch
+
+mel_step_size = 16
+
+def _load(checkpoint_path):
+    # Handle loading for different devices
+    checkpoint = torch.load(checkpoint_path, map_location=torch.device(device))
+    return checkpoint
+
+
+def main(face, audio, model, slow_mode=False):
+	if slow_mode:
+		print("Using SLOW animation mode (full face animation)")
+	else:
+		print("Using FAST animation mode (lips only)")
+		
+	if not os.path.isfile(face):
+		raise ValueError('--face argument must be a valid path to video/image file')
+
+	elif face.split('.')[1] in ['jpg', 'png', 'jpeg'] and not slow_mode:
+		full_frames = [cv2.imread(face)]
+		fps = args.fps
+
+	else:
+		video_stream = cv2.VideoCapture(face)
+		fps = video_stream.get(cv2.CAP_PROP_FPS)
+		
+		# Get video dimensions for potential downscaling of large videos
+		frame_width = int(video_stream.get(cv2.CAP_PROP_FRAME_WIDTH))
+		frame_height = int(video_stream.get(cv2.CAP_PROP_FRAME_HEIGHT))
+		total_frames = int(video_stream.get(cv2.CAP_PROP_FRAME_COUNT))
+		
+		# Auto-adjust resize factor for very large videos
+		original_resize_factor = args.resize_factor
+		if frame_width > 1920 or frame_height > 1080:
+			# For 4K or larger videos, use a higher resize factor
+			if frame_width >= 3840 or frame_height >= 2160:
+				args.resize_factor = max(4, args.resize_factor)
+				print(f"Auto-adjusting resize factor to {args.resize_factor} for high-resolution video")
+			# For 1080p-4K videos
+			elif frame_width > 1920 or frame_height > 1080:
+				args.resize_factor = max(2, args.resize_factor)
+				print(f"Auto-adjusting resize factor to {args.resize_factor} for high-resolution video")
+
+		print('Reading video frames...')
+
+		full_frames = []
+		
+		# For large videos, report progress and limit memory usage
+		frame_limit = 5000  # Maximum number of frames to process at once
+		if total_frames > frame_limit:
+			print(f"Large video detected ({total_frames} frames). Will process in chunks.")
+		
+		# Use tqdm for progress reporting
+		pbar = tqdm(total=min(total_frames, frame_limit))
+		frame_count = 0
+		
+		while frame_count < frame_limit:
+			still_reading, frame = video_stream.read()
+			if not still_reading:
+				video_stream.release()
+				break
+				
+			if args.resize_factor > 1:
+				frame = cv2.resize(frame, (frame.shape[1]//args.resize_factor, frame.shape[0]//args.resize_factor))
+
+			if args.rotate:
+				frame = cv2.rotate(frame, cv2.cv2.ROTATE_90_CLOCKWISE)
+
+			y1, y2, x1, x2 = args.crop
+			if x2 == -1: x2 = frame.shape[1]
+			if y2 == -1: y2 = frame.shape[0]
+
+			frame = frame[y1:y2, x1:x2]
+
+			full_frames.append(frame)
+			frame_count += 1
+			pbar.update(1)
+			
+			# For very large videos, limit frames to avoid memory issues
+			if frame_count >= frame_limit:
+				print(f"Reached frame limit of {frame_limit}. Processing this chunk.")
+				break
+		
+		pbar.close()
+		
+		# Reset resize factor to original value after processing
+		args.resize_factor = original_resize_factor
+
+	print ("Number of frames available for inference: "+str(len(full_frames)))
+
+	if not audio.endswith('.wav'):
+		print('Extracting raw audio...')
+		command = 'ffmpeg -y -i {} -strict -2 {}'.format(audio, 'temp/temp.wav')
+
+		subprocess.call(command, shell=True)
+		audio = 'temp/temp.wav'
+
+	wav = load_wav(audio, 16000)
+	mel = melspectrogram(wav)
+	print(mel.shape)
+
+	if np.isnan(mel.reshape(-1)).sum() > 0:
+		raise ValueError('Mel contains nan! Using a TTS voice? Add a small epsilon noise to the wav file and try again')
+
+	mel_chunks = []
+	mel_idx_multiplier = 80./fps 
+	i = 0
+	while 1:
+		start_idx = int(i * mel_idx_multiplier)
+		if start_idx + mel_step_size > len(mel[0]):
+			mel_chunks.append(mel[:, len(mel[0]) - mel_step_size:])
+			break
+		mel_chunks.append(mel[:, start_idx : start_idx + mel_step_size])
+		i += 1
+
+	print("Length of mel chunks: {}".format(len(mel_chunks)))
+
+	full_frames = full_frames[:len(mel_chunks)]
+
+	batch_size = args.wav2lip_batch_size
+	gen = datagen(full_frames.copy(), mel_chunks)
+
+	# Initialize video writer outside the try block
+	out = None
+	try:
+		for i, (img_batch, mel_batch, frames, coords) in enumerate(tqdm(gen, 
+												total=int(np.ceil(float(len(mel_chunks))/args.wav2lip_batch_size)))):
+			if i == 0:
+				#model = load_model(checkpoint_path)
+				print ("Model loaded")
+
+				frame_h, frame_w = full_frames[0].shape[:-1]
+				out = cv2.VideoWriter('wav2lip/temp/result.avi', 
+										cv2.VideoWriter_fourcc(*'DIVX'), fps, (frame_w, frame_h))
+
+			img_batch = torch.FloatTensor(np.transpose(img_batch, (0, 3, 1, 2))).to(device)
+			mel_batch = torch.FloatTensor(np.transpose(mel_batch, (0, 3, 1, 2))).to(device)
+
+			with torch.no_grad():
+				pred = model(mel_batch, img_batch)
+
+			pred = pred.cpu().numpy().transpose(0, 2, 3, 1) * 255.
+			
+			for p, f, c in zip(pred, frames, coords):
+				y1, y2, x1, x2 = c
+				p = cv2.resize(p.astype(np.uint8), (x2 - x1, y2 - y1))
+
+				f[y1:y2, x1:x2] = p
+				out.write(f)
+	except Exception as e:
+		print(f"Error during processing: {str(e)}")
+		print("Attempting to save any completed frames...")
+	
+	# Save the results - only if out was initialized
+	if out is not None:
+		out.release()
+	
+	# Convert the output video to MP4 if needed - only if the AVI exists
+	result_path = 'wav2lip/results/result_voice.mp4'
+	if os.path.exists('wav2lip/temp/result.avi'):
+		# Check if the result file is valid (has frames)
+		avi_info = os.stat('wav2lip/temp/result.avi')
+		if avi_info.st_size > 1000:  # If file is too small, it's likely empty
+			# Modified command to include the audio file
+			command = 'ffmpeg -y -i {} -i {} -c:v libx264 -preset ultrafast -c:a aac -map 0:v:0 -map 1:a:0 {}'.format(
+				'wav2lip/temp/result.avi', audio, result_path)
+			try:
+				subprocess.call(command, shell=True)
+				if os.path.exists(result_path):
+					print(f"Successfully created output video with audio at {result_path}")
+				else:
+					print(f"Error: Output video file was not created.")
+			except Exception as e:
+				print(f"Error during video conversion: {str(e)}")
+		else:
+			print(f"Warning: Output AVI file is too small ({avi_info.st_size} bytes). Face detection may have failed.")
+	else:
+		print("No output video was created. Face detection likely failed completely.")
+		# Return a default path even if no output was created
+	
+	# Return even if there were errors
+	return result_path

wav2lip/models/__init__.py

@@ -0,0 +1,2 @@
+from .wav2lip import Wav2Lip, Wav2Lip_disc_qual
+from .syncnet import SyncNet_color

wav2lip/models/conv.py

@@ -0,0 +1,44 @@
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+class Conv2d(nn.Module):
+    def __init__(self, cin, cout, kernel_size, stride, padding, residual=False, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.conv_block = nn.Sequential(
+                            nn.Conv2d(cin, cout, kernel_size, stride, padding),
+                            nn.BatchNorm2d(cout)
+                            )
+        self.act = nn.ReLU()
+        self.residual = residual
+
+    def forward(self, x):
+        out = self.conv_block(x)
+        if self.residual:
+            out += x
+        return self.act(out)
+
+class nonorm_Conv2d(nn.Module):
+    def __init__(self, cin, cout, kernel_size, stride, padding, residual=False, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.conv_block = nn.Sequential(
+                            nn.Conv2d(cin, cout, kernel_size, stride, padding),
+                            )
+        self.act = nn.LeakyReLU(0.01, inplace=True)
+
+    def forward(self, x):
+        out = self.conv_block(x)
+        return self.act(out)
+
+class Conv2dTranspose(nn.Module):
+    def __init__(self, cin, cout, kernel_size, stride, padding, output_padding=0, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.conv_block = nn.Sequential(
+                            nn.ConvTranspose2d(cin, cout, kernel_size, stride, padding, output_padding),
+                            nn.BatchNorm2d(cout)
+                            )
+        self.act = nn.ReLU()
+
+    def forward(self, x):
+        out = self.conv_block(x)
+        return self.act(out)

wav2lip/models/syncnet.py

@@ -0,0 +1,66 @@
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from .conv import Conv2d
+
+class SyncNet_color(nn.Module):
+    def __init__(self):
+        super(SyncNet_color, self).__init__()
+
+        self.face_encoder = nn.Sequential(
+            Conv2d(15, 32, kernel_size=(7, 7), stride=1, padding=3),
+
+            Conv2d(32, 64, kernel_size=5, stride=(1, 2), padding=1),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(64, 128, kernel_size=3, stride=2, padding=1),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(128, 256, kernel_size=3, stride=2, padding=1),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(256, 512, kernel_size=3, stride=2, padding=1),
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(512, 512, kernel_size=3, stride=2, padding=1),
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=0),
+            Conv2d(512, 512, kernel_size=1, stride=1, padding=0),)
+
+        self.audio_encoder = nn.Sequential(
+            Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(32, 64, kernel_size=3, stride=(3, 1), padding=1),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(64, 128, kernel_size=3, stride=3, padding=1),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(128, 256, kernel_size=3, stride=(3, 2), padding=1),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(256, 512, kernel_size=3, stride=1, padding=0),
+            Conv2d(512, 512, kernel_size=1, stride=1, padding=0),)
+
+    def forward(self, audio_sequences, face_sequences): # audio_sequences := (B, dim, T)
+        face_embedding = self.face_encoder(face_sequences)
+        audio_embedding = self.audio_encoder(audio_sequences)
+
+        audio_embedding = audio_embedding.view(audio_embedding.size(0), -1)
+        face_embedding = face_embedding.view(face_embedding.size(0), -1)
+
+        audio_embedding = F.normalize(audio_embedding, p=2, dim=1)
+        face_embedding = F.normalize(face_embedding, p=2, dim=1)
+
+
+        return audio_embedding, face_embedding

wav2lip/models/wav2lip.py

@@ -0,0 +1,184 @@
+import torch
+from torch import nn
+from torch.nn import functional as F
+import math
+
+from .conv import Conv2dTranspose, Conv2d, nonorm_Conv2d
+
+class Wav2Lip(nn.Module):
+    def __init__(self):
+        super(Wav2Lip, self).__init__()
+
+        self.face_encoder_blocks = nn.ModuleList([
+            nn.Sequential(Conv2d(6, 16, kernel_size=7, stride=1, padding=3)), # 96,96
+
+            nn.Sequential(Conv2d(16, 32, kernel_size=3, stride=2, padding=1), # 48,48
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True)),
+
+            nn.Sequential(Conv2d(32, 64, kernel_size=3, stride=2, padding=1),    # 24,24
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True)),
+
+            nn.Sequential(Conv2d(64, 128, kernel_size=3, stride=2, padding=1),   # 12,12
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True)),
+
+            nn.Sequential(Conv2d(128, 256, kernel_size=3, stride=2, padding=1),       # 6,6
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True)),
+
+            nn.Sequential(Conv2d(256, 512, kernel_size=3, stride=2, padding=1),     # 3,3
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),),
+            
+            nn.Sequential(Conv2d(512, 512, kernel_size=3, stride=1, padding=0),     # 1, 1
+            Conv2d(512, 512, kernel_size=1, stride=1, padding=0)),])
+
+        self.audio_encoder = nn.Sequential(
+            Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(32, 32, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(32, 64, kernel_size=3, stride=(3, 1), padding=1),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(64, 128, kernel_size=3, stride=3, padding=1),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(128, 256, kernel_size=3, stride=(3, 2), padding=1),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+
+            Conv2d(256, 512, kernel_size=3, stride=1, padding=0),
+            Conv2d(512, 512, kernel_size=1, stride=1, padding=0),)
+
+        self.face_decoder_blocks = nn.ModuleList([
+            nn.Sequential(Conv2d(512, 512, kernel_size=1, stride=1, padding=0),),
+
+            nn.Sequential(Conv2dTranspose(1024, 512, kernel_size=3, stride=1, padding=0), # 3,3
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),),
+
+            nn.Sequential(Conv2dTranspose(1024, 512, kernel_size=3, stride=2, padding=1, output_padding=1),
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(512, 512, kernel_size=3, stride=1, padding=1, residual=True),), # 6, 6
+
+            nn.Sequential(Conv2dTranspose(768, 384, kernel_size=3, stride=2, padding=1, output_padding=1),
+            Conv2d(384, 384, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(384, 384, kernel_size=3, stride=1, padding=1, residual=True),), # 12, 12
+
+            nn.Sequential(Conv2dTranspose(512, 256, kernel_size=3, stride=2, padding=1, output_padding=1),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(256, 256, kernel_size=3, stride=1, padding=1, residual=True),), # 24, 24
+
+            nn.Sequential(Conv2dTranspose(320, 128, kernel_size=3, stride=2, padding=1, output_padding=1), 
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(128, 128, kernel_size=3, stride=1, padding=1, residual=True),), # 48, 48
+
+            nn.Sequential(Conv2dTranspose(160, 64, kernel_size=3, stride=2, padding=1, output_padding=1),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),
+            Conv2d(64, 64, kernel_size=3, stride=1, padding=1, residual=True),),]) # 96,96
+
+        self.output_block = nn.Sequential(Conv2d(80, 32, kernel_size=3, stride=1, padding=1),
+            nn.Conv2d(32, 3, kernel_size=1, stride=1, padding=0),
+            nn.Sigmoid()) 
+
+    def forward(self, audio_sequences, face_sequences):
+        # audio_sequences = (B, T, 1, 80, 16)
+        B = audio_sequences.size(0)
+
+        input_dim_size = len(face_sequences.size())
+        if input_dim_size > 4:
+            audio_sequences = torch.cat([audio_sequences[:, i] for i in range(audio_sequences.size(1))], dim=0)
+            face_sequences = torch.cat([face_sequences[:, :, i] for i in range(face_sequences.size(2))], dim=0)
+
+        audio_embedding = self.audio_encoder(audio_sequences) # B, 512, 1, 1
+
+        feats = []
+        x = face_sequences
+        for f in self.face_encoder_blocks:
+            x = f(x)
+            feats.append(x)
+
+        x = audio_embedding
+        for f in self.face_decoder_blocks:
+            x = f(x)
+            try:
+                x = torch.cat((x, feats[-1]), dim=1)
+            except Exception as e:
+                print(x.size())
+                print(feats[-1].size())
+                raise e
+            
+            feats.pop()
+
+        x = self.output_block(x)
+
+        if input_dim_size > 4:
+            x = torch.split(x, B, dim=0) # [(B, C, H, W)]
+            outputs = torch.stack(x, dim=2) # (B, C, T, H, W)
+
+        else:
+            outputs = x
+            
+        return outputs
+
+class Wav2Lip_disc_qual(nn.Module):
+    def __init__(self):
+        super(Wav2Lip_disc_qual, self).__init__()
+
+        self.face_encoder_blocks = nn.ModuleList([
+            nn.Sequential(nonorm_Conv2d(3, 32, kernel_size=7, stride=1, padding=3)), # 48,96
+
+            nn.Sequential(nonorm_Conv2d(32, 64, kernel_size=5, stride=(1, 2), padding=2), # 48,48
+            nonorm_Conv2d(64, 64, kernel_size=5, stride=1, padding=2)),
+
+            nn.Sequential(nonorm_Conv2d(64, 128, kernel_size=5, stride=2, padding=2),    # 24,24
+            nonorm_Conv2d(128, 128, kernel_size=5, stride=1, padding=2)),
+
+            nn.Sequential(nonorm_Conv2d(128, 256, kernel_size=5, stride=2, padding=2),   # 12,12
+            nonorm_Conv2d(256, 256, kernel_size=5, stride=1, padding=2)),
+
+            nn.Sequential(nonorm_Conv2d(256, 512, kernel_size=3, stride=2, padding=1),       # 6,6
+            nonorm_Conv2d(512, 512, kernel_size=3, stride=1, padding=1)),
+
+            nn.Sequential(nonorm_Conv2d(512, 512, kernel_size=3, stride=2, padding=1),     # 3,3
+            nonorm_Conv2d(512, 512, kernel_size=3, stride=1, padding=1),),
+            
+            nn.Sequential(nonorm_Conv2d(512, 512, kernel_size=3, stride=1, padding=0),     # 1, 1
+            nonorm_Conv2d(512, 512, kernel_size=1, stride=1, padding=0)),])
+
+        self.binary_pred = nn.Sequential(nn.Conv2d(512, 1, kernel_size=1, stride=1, padding=0), nn.Sigmoid())
+        self.label_noise = .0
+
+    def get_lower_half(self, face_sequences):
+        return face_sequences[:, :, face_sequences.size(2)//2:]
+
+    def to_2d(self, face_sequences):
+        B = face_sequences.size(0)
+        face_sequences = torch.cat([face_sequences[:, :, i] for i in range(face_sequences.size(2))], dim=0)
+        return face_sequences
+
+    def perceptual_forward(self, false_face_sequences):
+        false_face_sequences = self.to_2d(false_face_sequences)
+        false_face_sequences = self.get_lower_half(false_face_sequences)
+
+        false_feats = false_face_sequences
+        for f in self.face_encoder_blocks:
+            false_feats = f(false_feats)
+
+        false_pred_loss = F.binary_cross_entropy(self.binary_pred(false_feats).view(len(false_feats), -1), 
+                                        torch.ones((len(false_feats), 1)).cuda())
+
+        return false_pred_loss
+
+    def forward(self, face_sequences):
+        face_sequences = self.to_2d(face_sequences)
+        face_sequences = self.get_lower_half(face_sequences)
+
+        x = face_sequences
+        for f in self.face_encoder_blocks:
+            x = f(x)
+
+        return self.binary_pred(x).view(len(x), -1)