Depending on the image encoding the diff image might contain more information (entropy) and would thus need more data for the encoding.
A simple example would be a totally blank image which wouldn’t require a lot of information as you would be able to store the color and just repeat it for all pixels, while an image containing random noise would contain much more information and would thus also be larger in file size.
The difference should not be visible, if you don’t use any image encoding/compression but store the raw pixel values.