The GPU utilization gives for a specific time period the percentage of time one or more GPU kernel(s) were running on the device. If your script shows a low utilization, you could profile it and check where the bottlenecks are. Usually this indicates, that the GPU is “starving”, i.e. your script cannot provide the data fast enough, which might happen for e.g. a slow data loading compared to the model execution.



 lingvisa:

Another question regrading the 2nd table, why is the ‘GPU Memory Usage’ table is blank? I though it should shows the actual GPU memory usage, which is different from the gpu usage column in the first table?


The second row would show all processes using the device. If this information is empty, it could point towards permission issues on your system, so that nvidia-smi doesn’t get information about the running processes and thus cannot display the memory each one is using.



 lingvisa:

Also, I found this little code to print out the currently gpu usage, which shows the first 2 gpu is not used:


The output of the script seems to match the output of nvidia-smi (it seems to be calling it, so this would be expected) but it seems that the device ids changed this time.