Thank you. Good suggestion. Having debugged the code and watched the tensors, I have noticed that rand_grid is on the different GPU which affects y as well. By fixing that, the problem got solved.
I replaced self.vertex = torch.from_numpy(point_set).cuda().float() with self.vertex = torch.from_numpy(point_set).float(), and rand_grid = self.vertex[idx,:] with rand_grid = self.vertex[idx,:].cuda(). I also deleted the following lines:
if x.is_cuda: # MY CHANGE: adding ".cuda()"
                y = y.cuda() # MY CHANGE: adding ".cuda()"
outs.append(self.decoder[i](y))

The reason behind this is that DataParallel devides the input tensors to two sets and copies each set to a different GPU. So, we always need to be sure that the tensors which are not the input arguments of the forward function are at the devise where the input arguments are.