README.md

metadata

license: apache-2.0
datasets:
  - 012shin/fake-audio-detection-augmented
language:
  - en
metrics:
  - accuracy
  - f1
  - recall
  - precision
base_model:
  - MIT/ast-finetuned-audioset-10-10-0.4593
pipeline_tag: audio-classification
library_name: transformers
tags:
  - audio
  - audio-classification
  - fake-audio-detection
  - ast
model-index:
  - name: ast-fakeaudio-detector
    results:
      - task:
          type: audio-classification
          name: Audio Classification
        dataset:
          name: fake-audio-detection-augmented
          type: 012shin/fake-audio-detection-augmented
        metrics:
          - type: accuracy
            value: 0.9662
          - type: f1
            value: 0.971
          - type: precision
            value: 0.9692
          - type: recall
            value: 0.9728

AST Fine-tuned for Fake Audio Detection

This model is a binary classification head fine-tuned version of MIT/ast-finetuned-audioset-10-10-0.4593 for detecting fake/synthetic audio. The original AST (Audio Spectrogram Transformer) classification head was replaced with a binary classification layer optimized for fake audio detection.

Model Description

• Base Model: MIT/ast-finetuned-audioset-10-10-0.4593 (AST pretrained on AudioSet)
• Task: Binary classification (fake/real audio detection)
• Input: Audio converted to Mel spectrogram (128 mel bins, 1024 time frames)
• Output: Binary prediction (0: real audio, 1: fake audio)
• Training Hardware: 2x NVIDIA T4 GPUs

Training Configuration

{
    'learning_rate': 1e-5,
    'weight_decay': 0.01,
    'n_iterations': 10000,
    'batch_size': 8,
    'gradient_accumulation_steps': 8,
    'validate_every': 500,
    'val_samples': 5000
}

Dataset Distribution

The model was trained on 012shin/fake-audio-detection-augmented dataset with the following class distribution:

Training Set (80%):
- Fake Audio (0): 43,460 samples (63.69%)
- Real Audio (1): 24,776 samples (36.31%)

Test Set (20%):
- Fake Audio (0): 10,776 samples (63.17%)
- Real Audio (1): 6,284 samples (36.83%)

Model Performance

Final metrics on validation set:

• Accuracy: 0.9662 (96.62%)
• F1 Score: 0.9710 (97.10%)
• Precision: 0.9692 (96.92%)
• Recall: 0.9728 (97.28%)

Usage Guide

1. Environment Setup

First, clone the AST repository and install required dependencies:

# Clone AST repository and set up path
git clone https://github.com/YuanGongND/ast.git
import sys
sys.path.append('./ast')
cd ast

# Install dependencies
pip install timm==0.4.5 wget

# Required imports
import os
import torch
import torchaudio
import matplotlib.pyplot as plt
import numpy as np
from torch import nn
from src.models import ASTModel

2. Model Implementation

Implement the BinaryAST model class:

class BinaryAST(nn.Module):
    def __init__(self, pretrained_path='pretrained_models/audioset_10_10_0.4593.pth'):
        super().__init__()
        # Initialize AST base model
        self.ast = ASTModel(
            label_dim=527,
            input_fdim=128,
            input_tdim=1024,
            imagenet_pretrain=True,
            audioset_pretrain=False,
            model_size='base384'
        )
        
        # Load pretrained weights if available
        if os.path.exists(pretrained_path):
            print(f"Loading pretrained weights from {pretrained_path}")
            state_dict = torch.load(pretrained_path, map_location='cpu', weights_only=True)
            self.ast.load_state_dict(state_dict, strict=False)

        # Binary classification head
        self.ast.mlp_head = nn.Sequential(
            nn.LayerNorm(768),
            nn.Dropout(0.3),
            nn.Linear(768, 1)
        )

    def forward(self, x):
        return self.ast(x)

3. Audio Processing Function

Function to preprocess audio files for model input:

def process_audio(file_path, sr=16000):
    """
    Process audio file for model inference.
    
    Args:
        file_path (str): Path to audio file
        sr (int): Target sample rate (default: 16000)
    
    Returns:
        torch.Tensor: Processed mel spectrogram (1024 x 128)
    """
    # Load audio
    audio_tensor, orig_sr = torchaudio.load(file_path)
    print(f"Initial tensor shape: {audio_tensor.shape}, sample_rate={orig_sr}")

    # Convert to mono if needed
    if audio_tensor.shape[0] > 1:
        audio_tensor = torch.mean(audio_tensor, dim=0, keepdim=True)

    # Resample to target sample rate
    if orig_sr != sr:
        resampler = torchaudio.transforms.Resample(orig_sr, sr)
        audio_tensor = resampler(audio_tensor)

    # Create mel spectrogram
    mel_spec = torchaudio.transforms.MelSpectrogram(
        sample_rate=sr,
        n_mels=128,
        n_fft=2048,
        hop_length=160
    )(audio_tensor)
    spec_db = torchaudio.transforms.AmplitudeToDB()(mel_spec)

    # Post-process spectrogram
    spec_db = spec_db.squeeze(0).transpose(0, 1)
    spec_db = (spec_db + 4.26) / (4.57 * 2)  # Normalize
    
    # Ensure correct length (pad/trim to 1024 frames)
    target_len = 1024
    if spec_db.shape[0] < target_len:
        pad = torch.zeros(target_len - spec_db.shape[0], 128)
        spec_db = torch.cat([spec_db, pad], dim=0)
    else:
        spec_db = spec_db[:target_len, :]

    return spec_db

4. Model Loading and Inference

Example of loading the model and running inference:

# Initialize and load model
model = BinaryAST()
checkpoint = torch.load('/content/final_model.pth', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Process audio file
spec = process_audio('path_to_audio.mp3')

# Visualize spectrogram (optional)
plt.figure(figsize=(10, 3))
plt.imshow(spec.numpy().T, aspect='auto', origin='lower')
plt.title('Mel Spectrogram')
plt.xlabel('Time Frames')
plt.ylabel('Mel Bins')
plt.colorbar()
plt.show()

# Run inference
spec_batch = spec.unsqueeze(0)
with torch.no_grad():
    output = model(spec_batch)
    prob_fake = torch.sigmoid(output).item()

print(f"Probability of fake audio: {prob_fake:.4f}")
print("Prediction:", "FAKE" if prob_fake > 0.5 else "REAL")

Key Notes:

• Ensure audio files are accessible and in a supported format
• The model expects 16kHz sample rate input
• Input audio is converted to mono if stereo
• The model outputs probability scores (>0.5 indicates fake audio)
• Visualization of spectrograms is optional but useful for debugging

Limitations

Important considerations when using this model:

1. The model works best with 16kHz audio input
2. Performance may vary with different types of audio manipulation not present in training data
3. Very short audio clips (<1 second) might not provide reliable results
4. The model should not be used as the sole determiner for real/fake audio detection

Training Details

The training process involved:

1. Loading the base AST model pretrained on AudioSet
2. Replacing the classification head with a binary classifier
3. Fine-tuning on the fake audio detection dataset for 10000 iterations
4. Using gradient accumulation (8 steps) with batch size 8
5. Implementing validation checks every 500 steps