This makes total sense and it is the case. This is what I am doing:
Coordinates === mapping function ===> Features  ====> model ====> output

The training loop is effectively from Features instead of Coordinates (these have requires_grad=True). The reason for that is because features are not being optimized and to avoid recomputing them on each epoch I just calculated them once. After your explanation, my mapping function has to be moved into model or the training loop so that inputs are really Coordinates and not Features. Is there any way I can make this work without mapping coordinates into features at every epoch?