Haha when I was young I thought that this effect was due to gravity, which is of course too weak for tiny objects to be observable. But it turns out that it is neither refraction nor diffraction nor parallax error. Instead it is due to having the wrong focus. If you are short-sighted like I am, then when looking at a far object each point will generate a circular disk image on your retina instead of a sharp point. When you move the edge of any object near your eye and it blocks part of the pupil, then the image generated on your retina will no longer be a full disk, hence the image appears to shift away from the edge. This explains your four later grid images. Notice that the rest of the grid is never in sharp focus, but the region near the edge of the occluding object is sharper, which is because those points generated less than a full disk on the camera sensor plane. As for your earlier images, they were due to the focus being beyond the object you were looking at. 530
As before, each point would result in a disk image on your retina, but inverted. Thus when another object blocks part of your pupil, the image appears to shift towards the edge instead of away from it.
To prove that this explanation is the correct one, put your face close to the screen without focusing on it, and move your finger in front of your eye. The image on the screen will appear to move towards your finger and also become sharper in the direction perpendicular to your finger's edge. Now repeat this experiment with the screen at a comfortable reading distance and make sure you focus exactly on the screen. Blocking your view now should have no effect on the apparent position of each pixel on the screen. If it still seems to move, it is because the blur (non-focused) image of your finger is interfering in an optical illusion. To prevent that, you can use a black thread instead. And if you focus somewhere in front of the screen, the image on the screen would appear to move away from your finger. If you use a black thread at an oblique angle to a grid on the screen, there will not be mismatching grid lines on the two sides of the thread if you focus exactly on the screen, but if you don't focus exactly on the screen the grid lines will indeed mismatch. 530