I will reply myself:
in the model forward, the correct code is:
def forward(self, index):
  # temporary affine transform matrix, no more within no_grad
  cm = torch.zeros(1,2,3, dtype=torch.double, device=self.mytensor.device, requires_grad=True)
  cm = cm + 0.0 ### here is the change
  cm[0,0,0], cm[0,1,1] = self.magfactor[0], self.magfactor[0]
  cm[0,:,2] = self.offval[index]
  # affine transform matrix is ready for actual forward pass  
  grid = torch.nn.functional.affine_grid(cM, self.mytensor.size())
  y = torch.nn.functional.grid_sample(self.mytensor, grid)
  # some operations on y
  retval = y**2
  return retval