You probably want to decide how you want to work on the audio (the signal itself, convolutions, spectrograms, mfccs).  You could manipulate the audio a bit doing small pitch changes, add noise, volume changes, etc to “shuffle” the data.  I would look into how data is loaded into text classification models / seq2seq models and then try to manipulate your audio loading accordingly.