This would most likely point to the data loading pipeline using a lot of memory.
If I understand this loader 1 correctly, the wav files are all preloaded and just indexed in the __getiteim__ method?
If so, try to use lazy loading and try to move the actual data loading into the __getitem__ method while the __init__ method gets the paths etc.