Do you have some code that we can examine? I put together a very simple program with multiprocessing and tried changing one of the module names during execution, but there was no crash. Are you potentially loading the module within the Dataset or Dataloader classes? Given that the program code is compiled into Python bytecode, and only then interpreted, I’m not sure why changing the Python code mid-execution will affect the program since it’s just the bytcode being interpreted. In fact, even deleting the bytecode during execution shouldn’t make a difference since the program has been loaded into memory. Any thoughts on what could cause this behavior @mrshenli?
As an aside, I would recommend checkpointing (and potentially torchelastic) so you don’t lose training progress for long-running jobs.