Took a closer look at my dataset. It had audio files which produced very long inputs which were the cause of OOM.
Lessons learned:

Check your data. Even if dataset description says, that it’s clean, prepared for training.
Audio sample rate matters - it should 8000 or 16000 (in my case). I had varying from 22.4kHz to 44.1kHz