Hi, thank you very much.
Say I have a sentence consisting of two words: S = [“Definitely”, “not”], and what I want is to transfer S into an embedding matrix T with a size of (2, 100), where each row represents a word.
I want to adopt BERT embeddings. But in BERT, each word is represented as a sub-word unit. This means that S will be represented as [“Def”, “##in”, “##ite”, “##ly”, “not”] ( “Definitely” is tokenized as “Def”, “##in”, “##ite”, “##ly”). BERT will output an embedding matrix H with a size of (5, 100) :(.
My goal is to merge some rows of H according to the sub-word units.
For example, for “Definitely”, I should merge the embeddings of [“Def”, “##in”, “##ite”, “##ly”] to get its presentation.
In my current method, I use a head mask vector h = [1, 0, 0, 0, 1] to record the “head” of each word, where 1 indicates the head position:
h = [
1, -> “Def”
0, -> “##in”
0, -> “##ite”
0, -> “##ly”
1 -> “not”
]
So I should merge rows which have a head mask of 0 to that having a head mask of 1. I have to use the for  computation to enumerate each element in  h , which is slow and can not batchfy.
Could you give me some efficient method to do the above computation?