Hello @wanchaol,
Thanks a lot for the quick reply!
I failed to mention this since I have had many issues over the course of this parallelisation journey of mine but I have tried this and I got the following error:
[W python_anomaly_mode.cpp:104] Warning: Error detected in torch::autograd::AccumulateGrad. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
    return self._bootstrap(parent_sentinel)
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/george/anaconda3/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 25, in main
    ddp_model = DDP(my_net(s).to(rank), device_ids=[rank], output_device=rank, find_unused_parameters=True)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 587, in __init__
    self._ddp_init_helper(parameters, expect_sparse_gradient, param_to_name_mapping)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 632, in _ddp_init_helper
    self.reducer = dist.Reducer(
 (function _print_stack)
Traceback (most recent call last):
  File "/home/george/edge ai/train_googlenet.py", line 173, in <module>
    data_parallel()
  File "/home/george/edge ai/train_googlenet.py", line 70, in data_parallel
    mp.spawn(
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/george/edge ai/train_googlenet.py", line 50, in main
    loss.backward()
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/george/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 186 has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration. You can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further debugging.

When using a static graph I got the following
RuntimeError: Your training graph has changed in this iteration, e.g., one parameter is unused in first iteration, but then got used in the second iteration. this is not compatible with static_graph set to True.

So I assume this also leads to a dead end unless, I am missing something.
A possible solution might be hidden in this:
Please make sure model parameters are not shared across multiple concurrent forward-backward passes

Although I have no clue about which/ or how parameters are shared across the models since each forward backward pass is separate with each other at least in the code I implemented (not sure about the PyTorch backend but I assume some parameter sharing must be going on for the synch of gradients).
I will edit the original post to provide this information as well.
Thanks a lot for taking the time to answer,
George