I think it is not possible, at least for now. You can record the padding information for each image and load the padding info along with the image batch. Then you can recover the original image from the padding info. In this way, the CNN will not waste time and computation with the padded zeros in the image.
But if you recover the original image, the image samples in a batch may become variable size. You can not feed them to the CNN in a whole and you have to feed the image on by one to the CNN. Since you haven’t given any specific information about the input to the network. I can only give the above suggestions.