It shouldn’t be necessary to change anything in the dataset, as the input shouldn’t change the grequires_grad flag of your output.
If the simple model gives you the right answer for synthetic data, I would suggest to add your data pipeline step by step and observe at which point the computation graph gets detached.