I want to profile my entire training and eval pytorch code. I am using custom dataloaders (e.g. torchmeta library) and novel pytorch libraries (e.g. higher library) and I see very significant performance slow down from what other libraries reported (despite me using better GPUs e.g. I use v100 vs titan xp). They take 2.5 hours while mine is taking 16h or more.
Instead of sharing the code I want to profile the two ENTIRE scripts and pin point what is slowing things down when I compare the profilers output.
Unfortuantely, I see a lot of profilers and it hard to chose and what is worse is that most examples seem focused on profiling a specific model and not include the dataloader. For me the dataloader and the entire code is crucial.
These are the resources I’ve found:

  
      pytorch.org
  
  
    

PyTorch Profiler — PyTorch Tutorials 1.7.0 documentation 3



  
  
    
    
  
  


python -m cProfile -s cumtime meta_learning_experiments_submission.py > profile.txt

  
      
      GitHub
  
  
    

adityaiitb/pyprof2 2

PyProf2: PyTorch Profiling tool. Contribute to adityaiitb/pyprof2 development by creating an account on GitHub.


  
  
    
    
  
  



  
      
      pytorch-lightning.readthedocs.io
  
  
    

Performance and Bottleneck Profiler — PyTorch Lightning 1.0.4 documentation 1



  
  
    
    
  
  



  
    
    
    Understanding PyTorch Profiler autograd
  
  
    I’m currently using torch.autograd.profiler.profile() and torch.autograd.profiler.record_function() from PyTorch Profiler for profiling my GPU program. I get confused with the output result by using prof.key_averages().table(). The output is organized as follows: 




Name
Self CPU total %
Self CPU total
CPU total %
CPU total
CPU time avg
CUDA total %
CUDA total
CUDA time avg
Number of Calls

















I don’t fully understand these, e.g., the relationship and difference between Self CPU t…
  


which one is recommended for profiling the entire code so that it works even with the presence of GPU? is:
python -m cProfile -s cumtime meta_learning_experiments_submission.py > profile.txt  

the best way to do this
(btw profiling seems better than changing my code randomly until it speeds up)

cross-posted:

How to profiling ENTIRE pytorch code when GPUs are present?
SO How to profiling ENTIRE pytorch code when GPUs are present? 7