You can check yesno dataset implementation 124 for a general idea about how to build a custom dataset.
Regarding paths in csv, you can maybe create a preprocessed file which contains the audio files as tensors similar to how they have done it in here 40.
Well a simple trick is to do you own calculations and set your own length in the __init__ of you dataset class, and return that length in the __len__ method. And not worry about the index argument. So the dataloader will call your __getitem__ length-1 times and you can randomly pick the frames you want from your data, irrespective of the index value.
As for trimming the audio, normalizing, stft calculation, etc, you can use any of the available transforms 21, or write your own transforms which you can pass to your custom dataset.