Deleting position IDS when fine-tuning BERT
Hi, 

I want help regarding this problem: 

I want to fine-tune BERT model on my custom dataset to fill in the missing words. So, I'm using run_language_modeling to do this. However, as my dataset contains sets of words and not real sentences, I don't really care about the order so I want to try deleting the position encoding by setting it to "zero tensor" to not affect the model. 

I would appreciate any help on how to add my costume position_IDS to run_language_modeling parameters? If it not possible to do so, is there any other way to fine-tune BERT with costume position_IDS? 

Thanks!  
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
