Dear all,
I develop a mathematical model based on PyTorch and utilize the torch.autograd method to provide gradient, Jacobian, and Hessian matrix to an optimization solver. I use cyipopt, which is a python wrapper of IPOPT.
When I test my snippet, the optimization step is very slow.
I profile my script with cProfile:
python -m cProfile -s 'tottime' hoge.py
and get the following outcomes:
         340915 function calls (332141 primitive calls) in 6.423 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       43    4.547    0.106    4.547    0.106 {method 'run_backward' of 'torch._C._EngineBase' objects}
    94/92    1.292    0.014    1.294    0.014 {built-in method _imp.create_dynamic}
      715    0.109    0.000    0.109    0.000 {method 'read' of '_io.BufferedReader' objects}
      715    0.044    0.000    0.044    0.000 {built-in method marshal.loads}
     3232    0.029    0.000    0.029    0.000 {built-in method posix.stat}

Apparently {method 'run_backward' of 'torch._C._EngineBase' objects} is a bottleneck.
Interestingly, when I profile the same script with:
python -m torch.utils.bottleneck hoge.py,
the optimization step requires a reasonable amount of CPU time, and indeed, the profiler returns:
--------------------------------------------------------------------------------
  cProfile output
--------------------------------------------------------------------------------
         80654 function calls (78746 primitive calls) in 0.160 seconds

   Ordered by: internal time
   List reduced from 945 to 15 due to restriction <15>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       26    0.018    0.001    0.018    0.001 {built-in method prod}
      180    0.014    0.000    0.014    0.000 {built-in method marshal.loads}
       50    0.012    0.000    0.012    0.000 {built-in method _imp.create_dynamic}
       43    0.011    0.000    0.011    0.000 {method 'run_backward' of 'torch._C._EngineBase' objects}
    50/45    0.006    0.000    0.017    0.000 {built-in method _imp.exec_dynamic}
        1    0.005    0.005    0.045    0.045 {method 'solve' of 'ipopt_wrapper.Problem' objects}

where {method 'run_backward' of 'torch._C._EngineBase' objects} is quick enough.
Where are the issues coming from? Why I can correctly execute the script with ‘torch.utils.bottleneck’ and not without the profiler? How can I correctly execute the script?
Thank you very much in advance.