Hey @Thunfisch, I am not familiar with RL, and cannot comment on the optimizer part, but I can share some pointers on using DDP/RPC for RL.
also cc @VitalyFedyunin for DataLoader/Sampler 
DDP applications usually have local loss tensors, and only gradients are synchronized during the backward pass. If you need to compute a global loss, you can combine DDP with RPC, using DDP to synchronize gradients and using RPC to calculate global loss and properly propagate gradients back to each DDP instance. Here are pointers:

RPC RL example: Implementing Batch RPC Processing Using Asynchronous Executions — PyTorch Tutorials 1.7.1 documentation 12

Combine DDP with RPC: Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.7.1 documentation 8