This solved the problem!
Just needed to change this:
x = spec(x)
x = torchaudio.functional.complex_norm(x)
x = mel(x)
x = log(x)

To this:
eps = 1e-9
x = spec(x)
x = torchaudio.functional.complex_norm(x + eps)
x = mel(x)
x = log(x)

I wasn’t sure if log(x) will cause a problem if x contains zeroes, so I added eps to it as well:
eps = 1e-9
x = spec(x)
x = torchaudio.functional.complex_norm(x + eps)
x = mel(x)
x = log(x + eps)