Separator token for custom QA input (multi paragraph, longer than 512)
Hello!
I'm trying to extract features for a QA task where the document is composed of multiple disparate paragraphs. So my input is:
question ||| document
where document is {para1 SEP para2 SEP para3 SEP}, so overall, it's something like:
question ||| para1 SEP para2 SEP para3 SEP

My question is: Is it okay to use the default BERT [SEP] token for the paragraph separation token as above? Or should I use something like the NULL token instead, or simply remove the paragraph separation token completely?

Secondly, my input is longer than 512, so I'm thinking of doing sliding windows like:
question ||| doc[:512]
question ||| doc[256:768]
and so on, finally merging the overlaps by averaging. Would this be correct?

Thanks!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
