Yes, you should be able to use your custom model as the backbone, at least I don’t think there should be any fundamental limitations to it as long as all shapes match the overall workflow.
I haven’t checked the code deeply, but e.g. this 59 might be a good place to use your custom model.