Work isn't just the force times the distance, it's the dot product of the vector force times the vector displacement, $$Work=\vec F \cdot \vec x$$ At the point of contact, the cylinder isn't sliding along the surface, so the only possible motion is perpendicular to the surface, and so the only possible motion at that point is perpendicular to the direction of the force.
Edit:
If the point of contact was moving away or towards the surface, that would mean that the cylinder itself was moving away or towards the surface.
In fact, the point on the edge is at rest, just for an instant. It's accelerating, so it doesn't stay at rest, but at the instant of contact the point is nonetheless at rest.
It's a little bit like how when something is thrown straight upwards that it's at rest for an instant at the top of its path. It doesn't stop dead, but it does stop for an instant while reversing its direction of motion.
Edit: a graphical explanation
The velocity of points on the the edge of a spinning disk go like.
A rolling disk, on the other hand adds in the velocity vector for the motion of the center of mass.
Edit: Here's the mathematical explanation for the motion of a point on the edge of a cylinder:
If we start with a spinning cylinder with a center of mass at rest we can describe a point on the outer surface of that cylinder by using a circle being swept out at a constant rate of rotation.
$$\vec x=r(cos(\omega t), -sin(\omega t)) $$ (The negative is because we are moving clockwise.) Taking the derivative, we get the velocity,
$$\vec v=r \omega (-sin(\omega t), -cos(\omega t)) $$ Since $v=r \omega $ $$\vec v=v (-sin(\omega t), cos(\omega t)) $$ If we add a constant velocity of $$\vec v_{along}=(v,0) $$ we get $$\vec v=(v,0)+v (-sin(\omega t), -cos(\omega t)) $$
The point on the bottom is given when $$\vec x=r(cos(\omega t), -sin(\omega t))=(0,-r) $$ So $sin(\omega t)=1$ and so, $$\vec v=(v,0)+v(-1,0)=(0,0) $$ 544