The training becomes slower. Anyone has any idea on the issues?


speed vs iteration
image912×582 23 KB


the grad graph size is consistent. The conclusion is drawn by calling the following function (input argument is loss.grad_fn) to see if there is more node added in the graph. The output here is always 11.


 def calc_num_node_in_grad_fn(grad_fn):
      result = 0
      if grad_fn is not None:
          result += 1
          if hasattr(grad_fn, 'next_functions'):
              for f in grad_fn.next_functions:
                  result += calc_num_node_in_grad_fn(f)
      return result



memory cost is not increased. nvidia-smi is called every 30 minutes. The memory size cost is always consistent.


pytorch version: 1.4.


Thanks very much