Hi,
this issue is still occurring but I found a workaround for now and I’m posting it here in case it will be useful to someone else -
apparently, librosa.load() is quite slow, and when I switched to scipy.io.wavfile.read() I got more than 10X speedup! since the load operation was 90% of the __getitem__ method, I can now run with num_workers = 0 and the GPU is fully loaded all the time.
however, I still find the behavior mentioned earlier to be strange…