The totality of light that fills space, even in a geometrical optics approximation, is a complicated light field that is difficult to visualize in full. Each point in space generally receives rays of various colors and from various directions simultaneously. We do not perceive or measure the light field unless something is placed in it to absorb or scatter the light -- ultimately our retinas.
Many surfaces that interact with light are not strongly sensitive to the direction from which it comes. These include matte (diffuse) screens, photographic film/CCDs, and our retinas (but not mirrors). At each point, the response effectively integrates rays over all incoming directions.
At a random point in space, this is likely to work out to a general ambient illumination that does not select for where the light originally came from -- i.e., does not form an image. Forming an image requires a specially manipulated light field (often, as in the eye, using a lens) that counteracts the tendency of bundles of rays to diverge and mix.
Now, we can clarify the pinhole situation (diagram inspired by seeking to clarify and correct pwf's answer). A pinhole is designed to form an image on a surface (screen or film) without the need for a lens. 530
This is accomplished by restricting the light field without bending any rays. The image is, of course, inverted (relative to the actual objects).
When your eye is looking directly through the pinhole, an image will form on the retina. The lens of the eye has makes relatively little difference in this case, assuming your focus is relaxed enough to look "through" the hole rather than "at" the hole. The image on the retina is inverted (relative to the actual objects) just as if you were looking at the same scene without any barrier. Therefore, your perceived view of the objects is non-inverted.
If you place a screen (not shown in diagram) facing the pinhole, it will show the inverted image. However, this is visually observed via diffusely reflected/transmitted light from the screen that forms its own light field. This light field is inverted relative to the original light field because its source (the pattern on the screen) is inverted relative to the original objects.
You perceive what is on the screen when your eye forms an image from this new light field. Therefore, the image of the screen that is formed on your retina is non-inverted (relative to the actual objects), and perceived as inverted. 530