I think your approach of initializing the embedding layers randomly and retrain them makes sense.
Could you try to use the strict=False argument when loading the state_dict via:
model.load_state_dict(state_dict, strict=False)

This should skip the mismatched layers.