Solved by JuanFMontesinos in post #12 
              
              
                Seems that this happens due to this line of code over a complex spectrogram: 
sp_mix = sp_mix_raw[:, ::2, ...]  # BxFxTx2

which can be “solved” by doing the slicing in cpu 
sp_mix = sp_mix_raw.cpu()[:, ::2, ...].cuda()  # BxFxTx2

It seems it collapse as memory is not contiguous soo another way of …