I am using pytorch 0.4.0, and I ran a simple training script with multiple GPUs in a cluster (managed by slurm).
I used cProfile to profile my code, and I found that tensor.item() function takes a lot of time!
ncalls tottime percall cumtime percall filename:lineno(function)
6868   685.9   0.09986 685.9   0.09986 ~:0(<method 'item' of 'torch._C._TensorBase' objects>)
5254   185.7   0.03535 185.7   0.03535 ~:0(<built-in method posix.read>)
1012   169.4   0.1674  169.4   0.1674  ~:0(<method 'cuda' of 'torch._C._TensorBase' objects>)
202    90      0.4456  90      0.4456  ~:0(<method 'type_as' of 'torch._C._TensorBase' objects>)
404    54.53   0.135   54.53   0.135   ~:0(<built-in method zeros>)
2      48.64   24.32   63.47   31.73   data.py:111(__init__)
202    44.23   0.219   44.23   0.219   ~:0(<built-in method torch._C._dist_all_gather_multigpu>)

As you can see, function “item()” takes almost half of the running time, and it is also slow in terms of percall. That is strange, because I think “item()” is a pretty simple function.
Is there any idea about possible reasons or solutions for this problem? May it be caused by distributed running? Or may it be caused by I always call this function with .cuda tensors?
Thank you for your help!