Hi,
I want to modify the BERT embedding Module (from pytorch-transformers package) in the way that I can dynamically add a tensor ontop of the tokenembedding.  The tensor contains additionally information that can’t be stored in an embedding layer, because it isnt dependend on the tokenposition or the token_id/vocab. I think the picture attached in the appendix describes my interest quite well (in red writing it is clarified how I would like to modify it).

modify.png888×584 17.7 KB