Decoding Large Audio Files Using Wav2Vec2ForCTC Model

https://discuss.huggingface.co/t/decding-large-audio-files-using-wav2vec2forctc-model/11097

I’ve been working on Wav2Vec2ForCTC model for a while. I used to have small audio files, i.e., audio files with relatively short durations (~ 1 min). When I tested the model on a large file (~ 14 mins), the model could not handle it in GPU, so, I shifted to use CPU. I notices that it used more than 200 GB of RAM to decode! I’ve tried to split the audio file into smaller audio files and use hidden states to link them together while decoding each segment but I could not find a way to feed the hidden states of the current audio file to the next audio file for the model to use it while decoding!

Any ideas or suggestions?
@patrickvonplaten, @anton-l

Hi @farisalasmary!

Unfortunately, there's no good way to use Wav2Vec-type models on large audio clips. Since it uses a bidirectional transformer as a context encoder, you can't feed hidden states between sequences, like in the autoregressive models.
But if you split the input into chunks with some additional padding (for better context), you can then concatenate the output character ids and decode them in one go, like so:

```python
import torch
import librosa
from transformers import AutoModelForCTC, Wav2Vec2Processor

device = "cuda"
model_path = "facebook/wav2vec2-base-960h"
sample_rate = 16000

model = AutoModelForCTC.from_pretrained(model_path).to(device)
processor = Wav2Vec2Processor.from_pretrained(model_path)

audio, _ = librosa.load("long_audio_file.wav", sr=sample_rate)
chunk_duration = 5 # sec
padding_duration = 1 # sec

chunk_len = chunk_duration*sample_rate
input_padding_len = int(padding_duration*sample_rate)
output_padding_len = model._get_feat_extract_output_lengths(input_padding_len)

all_preds = []
for start in range(input_padding_len, len(audio)-input_padding_len, chunk_len):
    chunk = audio[start-input_padding_len:start+chunk_len+input_padding_len]
    
    input_values = processor(chunk, sampling_rate=sample_rate, return_tensors="pt").input_values
    with torch.no_grad():
        logits = model(input_values.to(device)).logits[0]
        logits = logits[output_padding_len:len(logits)-output_padding_len]
            
        predicted_ids = torch.argmax(logits, dim=-1)
        all_preds.append(predicted_ids.cpu())

transcription= processor.decode(torch.cat(all_preds))
```

Note that this snippet isn't well-tested and could contain some off-by-one bugs, but it should give you the general idea  :slightly_smiling_face: 
However, we are working on integrating a native streaming inference solution into `transformers`, stay tuned for future updates! <|||||>Hi, @anton-l
Thank you for your reply! 
I'll try to use your approach and see what will happen. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>hey @anton-l, thank you for your code, it doesn't handle the first and last chunks properly though, do you think its better to add empty audio chunks in the beginning and the end of an audio or just not use start padding in the 1st chunk and end padding in the last chunk?