Ok, what do you think about loading one model, copying the pre-trained weights, initializing all other weights and finally saving this model as your baseline.
Then you would have to rewrite the initialize_weights function and load all weights from this baseline model.