Roj:

’m already wrapping the individual training calls in a function; could you post an example of what you did differently?


I have lots of wrappers and don’t use global variables. Runner looks like:
class Runner:
  def inner(...):
    datasource = create_datasource(...)
    trainable = create_trainable(datasource,...)
    return trainable.run(...)

  def run():
    try:
      r=self.inner(...)
    except Exception as ex:
      import traceback
      try:
        traceback.clear_frames(sys.last_traceback)
      except:
        pass
      raise ex
    gc.collect()
    torch.cuda.empty_cache()
    self.logger.debug("cuda_allocated: %d Mb", torch.cuda.memory_allocated() // (1 << 20))      

“trainable” encapsulates everything: torch module, optimizer, train loop (ignite).