Hello, lets get to your points one by one

DDP and DP is slow


My interpretation
Training on a single GPU takes about 2:50 minutes for all parameters except mixed precision, which increases speed to around 2 minutes.
So perfect parallelization would mean that the same time would be required with 4 GPUs if every single GPU gets a mini-batch with size 4, correct?
DataParallel seems to behave very similar to the hyperparameters, training takes around 5:25 mintues, except for mixed precision, which decreases it to 4:25 minutes.


Now to DistributedDataParallel:
Increasing the number of workers seems to slow down training by a lot.
Mixed precision has no effect on training speed (even though I observed on the GPUs that the required ram was decreased compared to not using it, and similar to the ram required for the mixed precision during DataParallel).

Your interpretation is correct, indeed, however, in python, whether you use thread based parallelism or process based parallelism, you are faced with extremely high costs:

for DP, it is mainly GIL cost
for DDP, is is mainly process communication cost.
The best way to increase efficiency is “batching”, because that part all happens in C/C++/CUDA domain, therefore, in order to fully display the power of DDP, you must make sure that your model is dealling with about 100800800*3 size of data (about 100 frames of images), even using this much data in a forward process on ResNet probably would take less than a second on powerful GPUs (eg: V100, your GPU), I am not sure how big your model prediction_module.protein_models it is, if it is not large enough, then batch=4 (16/4=4) per process probably is too small, and overhead of inter-process communication would demonstrate its annoying existence in this condition.

More workers, more slowly, it is true in python, whether you are using threads or processes, unless there is no communication overhead (for thread, it is GIL, for process, it is repeated serialization & deserialization, inter-process synchronization, etc.)
DistributedSampler in this case could also be a major overhead contributor, since internally the DataLoader will use _MultiProcessingDataLoaderIter, which uses a inter process queue to get data from sub-processes, if your data is not big enough and number is large, then it is very likely that repeated serialization-deserialization would contribute a lot to the slowiness, because for cpu tensors, they have to be moved to the shared memory first, then the handle will be serialized, there is no way for you to avoid this nightmare, if you are using the inbuit datasampler.
It is possible for you to customize your own dataloader, maybe load all data at once to your GPUs, (I believe V100 has enough memory to hold all of them, and you should have multiple V100s), however, it would require a huge quantity of time to debug your impementation.
In summary:
Maybe…, you should increase your batch size.
Live with this.
Make your own implementation.