There is  torch.cuda.profiler.profile() - and seems there is no documentation for it (though one can easily find source code)? wonder if it’s intentionally ‘hidden’? It works fine for me but only for 1 device (GPU)
At the same time can’t make torch.autograd.profiler.profile() working (with use_cuda=True in particular) - i.e. nvprof --profile-from-start off doesn’t profile anything (“No kernels were profiled.”). Also tried using emit_nvtx as per this example:
https://pytorch.org/docs/stable/_modules/torch/autograd/profiler.html#emit_nvtx 27
and it gives me :
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/profiler.py", line 309, in __enter__
    torch.autograd._enable_profiler(torch.autograd.ProfilerState.NVTX)
RuntimeError: can't change kind of profiling (e.g. NVTX to CPU) while profiler is running

Wonder if i’m doing smth wrong?
Separately (from user perspective) torch.cuda.profiler seem to be more natural namespace (as strictly speaking i might not have any models at all running, but rather measuring some tensor operations)
Thanks