Hi johank.
(I am not an expert in any of these areas.)
I don’t think GPT-2 has any knowledge of anything “Deep down”.   The way the model works is only probabilistic.   It doesn’t automatically “know” even simple things like sizes.  If you ask it how many pints to a gallon, it might be able to tell you, but it might also generate a paragraph that implies that a pint is bigger than a gallon, without “realising” that it should check for that kind of error.
I suppose, if GPT2 has seen enough descriptions of a “pointing finger” it might be able to associate a description with the label, but I don’t think that’s what you are after.
There is almost certainly a better understanding of  “pointing finger” inside your head than in GPT2.   Although you are having trouble thinking of how to define the labels during training, I think you would be better at it than GPT2.
If you “discard the first textual tokenization and embedding step”, then the whole trained GPT2 effectively becomes untrained.
When people make a “victory” gesture they mean something completely different to a “2” gesture.  If GPT2 “knows” about either or both of those gestures, it will “know” about them as completely different things.   It is unlikely that GPT2 will ever have been told that the two physical manifestations are similar.
When you say “gesture”, do you mean a single hand shape, or is the motion important?
I would be interested to see whether a neural network could learn to distinguish between “victory” and “2”.   Obviously, it can only learn to distinguish them if the training data has something different about them.  I imagine it might have, particularly if your data includes motion and not merely shape
GPT2 might possibly have some memory about what other text is commonly found in the vicinity of the words “pointing finger”.
You could test out what gestures GPT2 already “knows” about, by feeding it some starter text such as “He made a  gesture of …” and looking at the probabilities for the next words.
GPT2 is very (very) large, and would need a lot of time and a lot of data to train it.   I think a smaller model would be more suitable.
My guess is that you don’t want a pre-trained text model at all.   I could be wrong about that.