You could either try to load your state_dict into the backbone via accessing model.backbone and changing some keys in the state_dict if needed or alternatively create a new model as seen here 2 with your custom backbone.