Gradient checkpointing in Wav2Vec2
Hi,
Has anyone managed to fine-tune a **Wav2Vec2** model on long audio recordings which cannot fit into a GPU even with `batch_size=1`?  I tried out to set `gradient_checkpointing=true`, but it didn't help to solve the _CUDA Out of Memory Error_. Could it mean that gradient checkpointing does not work properly with **Wav2Vec2** models or are there other tricks needed to be added to the fine-tuning script in addition to the gradient checkpointing?
Hi! You can check out this related issue: https://github.com/huggingface/transformers/issues/10366<|||||>Thanks @LysandreJik! Yeah, one solution could be modify the `run_asr.py` script and add segmenting long speech samples by manually splitting each `batch["speech"]` into smaller chunks before passing to the _wav2vec2 processor_ and converting to input values, then passing one chunk at a time to the model and after that merging the outputs into a single transcription and calculating the loss. Just wondering if there could be any other ways / built-in features to split a batch of size 1 into smaller mini-batches.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @Getmany1  I also meet this error，But my audio file is no longer than 15s. How to solve the error? and how to pass one chunk at a time to the model.
