It’s not about shuffling. You can reduce the amount of RAM by reducing the amount of workers. Each worker loads a full batch, thus if you batch size is large and you have lot of workers you are linearlly increasing the ram needed.
There is a running time until dataloader reaches its nominal workflow. You cannot really test dataloader by doing next(iter(dataloader))
You can also use numpy memory map not to load the big files you mentioned all at once but on demand. Soundfile allows to read on demand too (wav files).