I think it depends how much “flexibility” you need and if you couldn’t achieve it by manipulating Huggingface models.
Rewriting the models sounds like a good way to get familiar with the implementations, but you should consider how long it might take, and check if this time would be better spent by manipulating the models and run more experiments.