Soo let me clarify.
What you say doesn’t make sense to me. You state that your data is fixed and that the sample is NaN before feeding the network. Those options are mutually exclusive. Either your dataloading pipeline has some randomness (for example, you are loading segments of audio randomly picked and you can get a full-zeros one) or your data is becoming NaN inside the network.
I would recommend you to check if the loss is inf or is NaN. If you backpropagate a NaN or an Inf, all the weights in the network will become NaN.
Are you reading some sort of data which can lead to issues? For example reading from buffers which may be empty, video readers which aren’t returning frames… don’t know.
Anyway you can add a simple check in the __getitem__ function looking for NaNs in your data and raising an exception with the path in case there is an issue. This way you can debug if the data is wrong or the dataloader is having troubles to read it.
Oh, a less likely option I saw (but no one could debug) is I was finding NaNs after allocating tensors on GPU for GTX 1080 model.  Sooo maybe another check (less likely one) is to check after and before allocating the tensors.