The position embedding in huggingface bert model is defined in the BertEmbedding class. Other models also have an Embedding class. If your task is aligned with what they have, you might be able to get away with changing the Embedding class only.

Thanks, @bengul, that looks like a very promising route. Will give it a try tomorrow!