There also are models working directly on waveforms, e.g. wav2letter 1.
For an even simpler entry, you might look at the Speech Commands tutorial 3 where the waveform is translated into class predictions for keywords (a much simpler problem and relatively fast to train).
Best regards
Thomas