Let’s say I have two batches of single-channel images, each of size 8x1x128x128. I am trying to apply spatial transformation to one batch to align it with another one (alignment measured with MSE loss).
This is what relevant part of my code looks like
# two batches of images are img1, img2

# Generating affine grid with Identity transformation
theta = torch.FloatTensor([1, 0, 0, 0, 1, 0])
theta = theta.view(2, 3)
theta = theta.expand(disp.size()[0],2,3)
identity_grid = F.affine_grid(theta,img1.size())

# disp is another variable of size Nx128x128x2, with values in [-1,1]
# net is the network I am using to generate displacements to each pixel
disp = net(img1)

new_grid = identity_grid + disp
# Apply spatial transformation to img1
img1t = F.grid_sample(img1,new_grid)

Lsim = nn.MSELoss()(img1t,img2)
Lsim.backward()

My aim is to backpropagate into parameters of net through disp. But when I do Lsim.backward(), new_grid.grad and disp.grad are both None.  What might be going wrong?