Hi,
I have figured out a way to edit the position embedding. The position embedding in huggingface bert model is defined in the BertEmbedding class. Other models also have an Embedding class. If your task is aligned with what they have, you might be able to get away with changing the Embedding class only. Look Here. Alternatively if you want to provide your preferred position embedding with the input, then you have to change all the classes that use the position embedding. I am not an expert by any means, but feel free to ask anything if you need more help on this.