If I’m not mistaken, fastai provides some kind of solution for this work flow.
Unfortunately, I cannot find the topic here (or am not sure which keywords were used in the discussion), but maybe @sgugger or @jphoward might give you more information.