Before forming the Tensor dataset I did have to perform some operations to construct the dataset in pandas but I don’t think that should affect autograd, right?

No that won’t
Can you share the traceback reported by the detect anomaly? Where does it point?