Hello, guys. I’ve some questions of image warping with optical flows.
I would like to test warp the images by grid_sample, and I’ve got samples(two images with flow) from PWC-Net as follows:
input image from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/data 94
flow file from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/tmp 82

test31024×436 459 KB

def warp(x, flo):
“”"
warp an image/tensor (im2) back to im1, according to the optical flow
x: [B, C, H, W] (im2)
flo: [B, 2, H, W] flow

"""
B, C, H, W = x.size()
# mesh grid
xx = torch.arange(0, W).view(1 ,-1).repeat(H ,1)
yy = torch.arange(0, H).view(-1 ,1).repeat(1 ,W)
xx = xx.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
yy = yy.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
grid = torch.cat((xx ,yy) ,1).float()

if x.is_cuda:
    grid = grid.cuda()
vgrid = Variable(grid) + flo

# scale grid to [-1,1]
vgrid[: ,0 ,: ,:] = 2.0 *vgrid[: ,0 ,: ,:].clone() / max( W -1 ,1 ) -1.0
vgrid[: ,1 ,: ,:] = 2.0 *vgrid[: ,1 ,: ,:].clone() / max( H -1 ,1 ) -1.0

vgrid = vgrid.permute(0 ,2 ,3 ,1)
flo = flo.permute(0 ,2 ,3 ,1)
output = F.grid_sample(x, vgrid)
mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
mask = F.grid_sample(mask, vgrid)

mask[mask <0.9999] = 0
mask[mask >0] = 1

return output*mask

test = cv2.readOpticalFlow(‘reference_frame_0010.flo’)
test = torch.from_numpy(test)
test = test.view(1, 2, H, W)
test = Variable(test).cuda()
test_img2 = cv2.imread(‘frame_0011.png’)
test_img2 = torch.from_numpy(test_img2)
test_img2 = test_img2.view(1, 3, H, W)
test_img2 = Variable(test_img2).cuda()
out = warp(test_img2, test)
out = out.view(H, W, 3)
out = out.cpu()
out = np.float32(out)
cv2.imwrite(‘test3.png’, out*255)
I’ve run the code above and got a result image. As you can see the last image, it is not I expected.
It seems like several part of the images with grid.
Is it right to get a warped target image with optical flow?
If not, how can I get whole image looks like the input color images?
Please, help me to solve this problem.