I believe per-parameter options should be supported by DistributedDataParallel. Have you tried it out and seen any issues? If you do see issues/unexpected behavior feel free to open an issue on github.