32 workers is probably way too much overhead. These workers are used for data loading. You’re probably better off with four-or-so workers. Forking might lead to an increase in memory.
If you want to use multiple GPUs you will definitely notice a speed up if you use DistributedDataParallel.