So thread 4 one seems to be a cuda driver thread. Not sure we can do anything about this ? (cc @ptrblck)
thread 1 is your main worker thread
thread 3/4 are autograd worker thread (not sure why there are 2), but the CPU worker thread should run only while thread 1 will be blocked on waiting for it to finish. So that should use more than one core.