Why did permanent marker come off of glass after rubbing it with my fingers? No, my fingers don't look stained but I accidentally made a stroke of permanent marker on glass and I was looking for nail polish remover to use, couldn't find it. So I started rubbing the stroke of permanent marker with my fingers and it started coming off. I was able to get it off with 3 fingers, with the thumb getting most of it off.

I wonder, is it because permanent marker dissolves in skin oil that I was able to get it off? Or was it the small amount of sweat? Or both? Or was it just that I was rubbing my fingers on the permanent marker and the shear force got it off?

I mean yes, this was just a few minutes after the accidental stroke that I got it off with my fingers but that would have been plenty of time for it to dry, I think.
Glass is a very smooth surface and therefore things adhere to it less effectively such as the ink in your permanent marker. Maybe your marker is not of a very good quality or maybe you didnt leave it long enough for the ink to stabilize on the surface.

Why dont you conduct an experiment to try your various hypotheses by varying one factor while keeping the rest constant? For exaple:

1) Try different brands of markers on the same glass surface leaving them to dry for the same time and then rubbing them in the same way.
2) Use the same marker but leave it to dry for different amount of time and then rub it in the same way each time.
3) Use the same marker on different surfaces and rub in the same way. 
4) Try with different amounts of dilute saline solution to simulate sweat or cooking oil to simulate skin oil etc.
etc.

Or even better: Try at glass surfaces that have been etched with different sandpaper grits.

Your experiment is reproducible though, i tried as well and small amounts come off easily.