Hello and thank you for the response.
When I say “Deep down” I mean what you say: that it might have some association between either close by gestures or in our case unseen gestures.
I was perhaps a bit unclear but the main goal is some kind of model that given a set of training gestures could predict new gestures. It is fairly straight forward to do classification which we successfully have done on a small set of gestures but if we would want to predict “general” or “new” gestures we need a different methodology. The large pre-trained textual transformers would be one interesting approach.
An example could be that we fine-tune with the gestures 0 to 9 and see if it could learn when we hold up all of our fingers. If we would in any way (paragraph or not) get an output that even alludes to 10 it is a great start. And I do not think it is completely unreasonable.
Gestures in our case can include the motion. What we read from our gloves is the relative angles of all joints. If we pack these in a sequence the motion is also captured. A start would be static gestures from a single hand.
Regarding the point of discarding the embedding step, I did a quick check with GPT2LMHeadModel in  and the following output was given with torchinfo on “gpt2” (i.e. smallest version):

GPT2LMHeadModel                          –
├─GPT2Model: 1-1                         –
│    └─Embedding: 2-1                    38,597,376
│    └─Embedding: 2-2                    786,432
│    └─Dropout: 2-3                      –
│    └─ModuleList: 2-4                   –
│    │    └─GPT2Block: 3-1               7,087,872
│    │    └─GPT2Block: 3-2               7,087,872
│    │    └─GPT2Block: 3-3               7,087,872
│    │    └─GPT2Block: 3-4               7,087,872
│    │    └─GPT2Block: 3-5               7,087,872
│    │    └─GPT2Block: 3-6               7,087,872
│    │    └─GPT2Block: 3-7               7,087,872
│    │    └─GPT2Block: 3-8               7,087,872
│    │    └─GPT2Block: 3-9               7,087,872
│    │    └─GPT2Block: 3-10              7,087,872
│    │    └─GPT2Block: 3-11              7,087,872
│    │    └─GPT2Block: 3-12              7,087,872
│    └─LayerNorm: 2-5                    1,536
├─Linear: 1-2                            38,597,376
Total params: 163,037,184
The embedding parameters are a large part of the model and you are probabably correct in that removing them might potentially throw away all the knowledge in the Blocks. Compared to e.g. changing the last linear layer, the embedding layer might be necessary to make use of what is stored in the Blocks. However I would be very interested in seeing if there are other similar approaches.
It is clear the the embedding layer is by no means the largest part of the model.
I think a pre-trained text model is necessary for the “deep down” knowledge. However, it is possible that there are other approaches. I have just become aware of Zero-shot learning which I will take a look at. But the usage of e.g. GPT-2 would still be very interesting (and very much in line with other awesome usages).