Solved by Martynas_Jurkus in post #16 
              
              
                Took a closer look at my dataset. It had audio files which produced very long inputs which were the cause of OOM. 
Lessons learned: 

Check your data. Even if dataset description says, that it’s clean, prepared for training.
Audio sample rate matters - it should 8000 or 16000 (in my case). I had var…