I would add a new dimension to those tensors and compare them:
all_idx_before = all_idx_before.view([size_batch, size_points, 20, 1])
idx = idx.view([size_batch, size_points, 1, 5])

absdiff = torch.abs(idx - all_idx_before) # [batch, points, 20, 5]

Then find indices with argmin:
output = absdiff.argmin(dim=2)