But its kind of distributed training where each of the neural network in different processes communicate while updating the weight

Does it have to communicate parameters instead of gradients? If all you need is to keep the parameters on all processes in sync, communicating gradients should be sufficient I think.

I don’t want to distribute a single set of input to multiple nodes/processes.

Yep, DDP does not split inputs, instead each process need to prepare its own inputs.
One question is, does the parameters/gradients communication occur in a synchronized or asynchronized fashion? “Synchronized” means all processes communicate at exactly the same time, while asynchronized can be something like gossip. DDP only works for synchronized use cases. If you need asynchronized communication, you can directly use c10d (allreduce, allgather, broadcast, etc.) and create multiple sub-groups to perform communicatioin.