There are cases where an image includes an unnecessary object, which is an object thought to be unnecessary by a user (such as facial blemishes or moles, or electric cables in the background). Conventionally, a function for performing inpainting processing after removing such an unnecessary object from the image has been proposed. For example, an image processing method has been proposed in which inpainting is performed by attaching the pixel values in a non-missing region to a missing region, which is a region from which an unnecessary object was removed. Specifically, patch sets including a missing patch, which is an image of a region that includes a missing region, and a reference patch, which is an image of a region that does not include a missing region, are provided, patch sets in which the missing patch and the reference patch are comparatively similar are selected, and then a patch set in which the missing patch image and the reference patch image are comparatively similar is selected based on the relationship between estimated pixel values in a missing region of the missing patch and the corresponding pixel values of the reference patch. The missing patch is then inpainted based on the reference patch in the patch set that was selected (e.g., see WO 2011/061943).