Is there any efficient way to convert BERT outputs to fit token-level tasks?
Say I have a sentence consisting of two words: S = [“Definitely”, “not”], and what I want is to transfer S into an embedding matrix T with a size of (2, 100), where each row represents a word.

I want to adopt BERT embeddings. But in BERT, each word is represented as a sub-word unit. This means that S will be represented as [“Def”, “##in”, “##ite”, “##ly”, “not”] ( “Definitely” is tokenized as “Def”, “##in”, “##ite”, “##ly”). BERT will output an embedding matrix H with a size of (5, 100) :(.

My goal is to merge some rows of H according to the sub-word units.
For example, for “Definitely”, I should merge the embeddings of [“Def”, “##in”, “##ite”, “##ly”] to get its presentation.

In my current method, I use a head mask vector h = [1, 0, 0, 0, 1] to record the “head” of each word, where 1 indicates the head position:
h = [
1, -> “Def”
0, -> “##in”
0, -> “##ite”
0, -> “##ly”
1 -> “not”
]
So I should merge rows which have a head mask of 0 to that having a head mask of 1. I have to use the `for` computation to enumerate each element in h, which is slow and can not batchfy.

Is there any efficient method to do the above computation?


i am also curious about it. ;;
https://github.com/dsindex/etagger/blob/master/feed.py
this is not efficient way. just pooling.<|||||>> i am also curious about it. ;;
> https://github.com/dsindex/etagger/blob/master/feed.py
> this is not efficient way. just pooling.

Hi, I think I get the solution. The problem can be solved via simple matrix computation.
For example, let the BERT representation of the above example be B, which has a size of (5, 100).  
We should first construct a matrix according to the sub-words:
m = 
[ 1, 1, 1, 1, 0
  0, 0, 0, 0, 1] 
Then we can simply compute m.dot(B), which is exactly the result.

<|||||>I have solved this issue.