I’ve written a pytorch script, and looking to speed it up.
I’ve tried the following:

use a c4.4xlarge, in cpu mode, instead of Mac OS X, in cpu mode => result is twice as slow as on Mac 

use an aws g2, in cuda mode => twice as fast as Mac laptop. yay
use an aws p2, in cuda mode => another 50% as fast as g2

Now at this point, I’m not sure which bits are slow

If it was a c++ script, that didnt use cuda, I might use either statistical sampling: ie, start in debugger, stop it, and store the stacktrace. do this eg 5-10 times, and look at which bits/functions tend to me in man yof the stacktraces => this is the bottleneck
if it was cltorch, or deepcl, well I pre-instrumented them with profilers 

in pytorch cuda, I suppose I should use an nviida profiler? Or … ?

Its not clear to me which bits of the program are taking the time, could be cpu bits, so I’d prefer to start at a higher level than nvidia profiler probably.  Thoughts on ideas / stadnard techinques, for profiling pytorch?