ptrblck:

Could you load the state_dict once on a machine with enough memory and then store parts of it to use the first approach?


This is the workaround I am currently implementing 
Thanks for your help, I will probably use this workaround for some time, things might change (for example, I might get some control over model saving code).