After investigating further I found the following topic 51, which led me to checking my dataloader, after which I found out that the error is indeed caused by my dataloader. Since I am using audio data I am making use of the torchaudio package. In my dataloader I load the audio, resample it and create a mel spectrogram on which I then train my model. Singling out the different preprocessing steps in my dataloader is caused by the creation of the mel spectrogram (torchaudio.transforms.MelSpectrogram). Since torchaudio is calling torchaudio.transforms.Spectrogram first and then creating the mel bins, I also tried using the spectrogram function. This continued normally for longer but still gave me the same error. See the following minimum working example:
import torch
import torchaudio

for epoch in range(10):
  print(f"Epoch {epoch+1}/{30}")
  for example in range(20000):
    if (example + 1) % 100 == 0:
      print(f"Example {example + 1}")
    x = torch.rand(1, 88200)
    mel_spec = torchaudio.transforms.MelSpectrogram()(x)

I am not sure if/how I can fix this or if the fix from the aforementioned topic can be applied to my case.
@ptrblck Do you happen to have any idea for what causes the problem/how I circumvent it?