Hi, I want to run two lines in parallel inside forward function on single GPU.
The two sub-processes are independent from each other. I want to run self.l1 and self.l2 simultaneously here. The minimum code is as follows:
import torch
import torch.nn as nn
import torch.multiprocessing as mp
import time

mp = mp.get_context('spawn')

class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.l1 = nn.Linear(784, 100)
        self.l2 = nn.Linear(784, 100)

    def forward(self, x):

        processes = []
        num_processes = 2

        for i in range(num_processes):
            if i == 0:
                p = mp.Process(target=self.l1(x))
            else:
                p = mp.Process(target=self.l2(x))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()

       

  
if __name__ == '__main__':
    BATCH_SIZE = 64
    x = torch.randn(BATCH_SIZE, 784).cuda()
    mynet = Net().cuda() # ; torch.cuda.synchronize()
    mynet(x)

I found similar question: for loop inside single GPU 23 and for loop inside forward 10.
There are two relevant questions : parallel over samples 7 and parallel execution 5, the answers to these two questions are reformulated version of the original problem.
Besides, I tried the nn.Parallel(block1, block2) inside this post two blocks in parallel 9 however it did not work.
I know distribute model to multiple GPU is well documented, however I have only one GPU here.
multiprocessing over model level is documented here multiprocessing over model level 13
Any help would be appreciated, thanks a lot!