I’ve had to deal with similar issues, you should feel lucky you only have 4 processes and not 64 
The fact that 3 are in 100% utilization means that are inside nccl sync operation like the one at the end of backward .backward(), while the 4th one is doing something non-GPU related, like waiting for user input.
The general strategy is to look at stack trace of the “odd one out” process. You can get C++ stack trace by doing "gdb -p " and “bt” or “thread apply all bt”
With a bit more work, you can get Python stack. This requires modifying client code. For instance, I run install_pdb_hander 77 on init of my processes. It allows me to break into PDB on CTRL+\ and look what the current process is doing.
When using distributed launcher, this will only send CTRL+\ to the launcher process, so for this to get you stack trace of arbitrary worker you could need to modify your launching procedure to run in 4 worker in 4 different tmux windows