Since this is master/client training scenario, it might fit into the parameter server paradigm. Please see this tutorial 3 and this example 3 for a complete example.

Can you tell how to make sure that each of the model has different input?

In this case, as each client is running in its own process, it can create its dedicated dataloader. Or if you need the master to dispatch the input data to those clients, this is also doable. You can wrap those client logic into a function and let the master to launch them, sth like  rpc.rpc_async("client_a", client_func, args=(inputs_for_a,)).

I need to have each of the model train separately on their own data.

Since the forward-backward does not need to span across machine boundaries, this RPC+RRef Example 2 might be sufficient, as you don’t need distributed autograd and distributed optimizer.