I forgot to update this thread. My main problem was the improper segmentation of some audio files. Once I did that correctly it ran just fine.
I’m fine-tuning the model to a very small amount of data so these numbers might probably not mean much to you (on a V100):

Batch size of 32
16kHz sampling rate
Mixed precision
~2mins per epoch