efficient method to fix isolated holes would be to prepare a substitute image, and do conditional blending with torch.where(hole, sub, original)
how to prepare such a substitute is another matter, using a smoothed image would be the simplest approach perhaps.