Also I have general question regarding multi processing in pytorch, if I have multi worker dataloader already, and I start my main training proces followling
https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices 9
In this case, the main process will spin off multi process, and data loader will also have multi process, will the dataloader and training share same processes underneath in OS? or both of them create new processes?
import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()