ptrblck:

On the other hand, GPU operations are already executed asynchronously, so in case the CPU operation has no dependency on the model etc. it would already be executed while the model is busy with the training on the GPU.


But the “main” thread is still busy in this case, i.e. it won’t multiplex CPU & GPU “command streams”. And I don’t think that blocking points for GPU sync. are “cooperative” with CPU ops (that are themselves blocking). So, I believe, additional thread/process is still needed for “off-model” CPU computations.