Jwin, I had the same problem and I solved using a sampling grid:
def downsampling(x, size=None, scale_factor=None, mode='nearest'):
	# define size if user has specified scale_factor
	if size is None: size = (int(scale_factor*x.size(2)), int(scale_factor*x.size(3)))
	# create coordinates
	h = torch.arange(0,size[0]) / (size[0]-1) * 2 - 1
	w = torch.arange(0,size[1]) / (size[1]-1) * 2 - 1
	# create grid
	grid = torch.zeros(size[0],size[1],2)
	grid[:,:,0] = w.unsqueeze(0).repeat(size[0],1)
	grid[:,:,1] = h.unsqueeze(0).repeat(size[1],1).transpose(0,1)
	# expand to match batch size
	grid = grid.unsqueeze(0).repeat(x.size(0),1,1,1)
	if x.is_cuda: grid = grid.cuda()
	# do sampling
	return F.grid_sample(x, grid, mode=mode)

You can either use nearest or bilinear interpolation, depending on your needs. In my case I was downsizing a semantic layout, so I couldn’t use any type of pooling (non-integer values wheren’t allowed and a cross integer value between two classes didn’t have sense).
I didn’t try this in upsampling. Notice that it only works with 4D tensors as input, so if you need to use it with a single image, before you have to .unsqueeze(0) it.