performance degraded when using paddings between queries and contexts.
I just want to ask this here and see whether other people encountered the same situation.

I am doing modifications on the run_squad.py example.

So for the original training feature, the input ids are [cls]qqqqq[sep]cccccc000000. The attention mask is just something like 111111100000 where first k inputs were masked with 1 and 0 for the laters.

I tried to generate input ids look like [cls]qqqq[sep]0000[sep]cccccccc00000, the I can have fixed length of query and context with proper padding. Also, I changed the attention mask accordingly, namely, to have something like 11110001111100000.

However, when I trained the model on this new feature, the  score degraded from 76 to 44 for the bertforquestionanswering model. I am wondering if there is any catastrophic results by doing this kind of masking? Did anyone experienced with similar situations?
Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?<|||||>> Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?

I felt like this is caused by the way how we pretrain BERT. BERT is pretrained on contiguous texts. The way we pad zeros in between broke such continuity so essentially we need to re-train the whole model. 

Since I was trying to separate query and context, the way I tackled this is just masking all query tokens to create the context and same for the query. <|||||>> > Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?
> 
> I felt like this is caused by the way how we pretrain BERT. BERT is pretrained on contiguous texts. The way we pad zeros in between broke such continuity so essentially we need to re-train the whole model.
> 
> Since I was trying to separate query and context, the way I tackled this is just masking all query tokens to create the context and same for the query.

It could be the same way as inputing the query and context separately if we mask the query or context part. Anyway, I will try it. Thanks.