@osalpekar Thanks a lot for your suggestion.
I had considered torch.multiprocessing.Pool.map since, as you correctly pointed out, seemed to be the solution.
Unfortunately, I see two issues with this (please find the adapted code below):
First: somehow pool.map seems to take quite some time (100x more than using a for-loop in this case, roughly 0.3 sec, which is around 3x longer than in my original problem). Do you know of any problems with this method? It seems strange, if you compare it with the built-in map function. Could it be some sort of overhead? seems quite high.
Second: Indeed your suggestion works, I made some small changes, but it doesn’t raise any errors, or freezes. However, this solution does not translate to the case where you have registered parameters in the functions f_i. It throws following error:
multiprocessing.pool.MaybeEncodingError: Error sending result
Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which 
requires_grad, since autograd does not support crossing process boundaries.  
If you just want to transfer the data, call detach() on the tensor before 
serializing (e.g., putting it on the queue).')'

So it is an issue with autograd. I have read this error already here 2 and since, it is relatively new to have distributed autograd, I’d need to dive into the docs. Any pointers are welcome!
I tried it in version 1.6.0 and 1.7.0.
Adapted code
import torch
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
import time

class DemoMultiProcModel(nn.Module):
    def __init__(self):
        super(DemoMultiProcModel, self).__init__()
        self.net1 = nn.Linear(196, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

        self.kernel = nn.Linear(4, 1) #let's try to make a feature map

    def func_to_apply(self, data):
        return self.kernel(data)
        # return torch.sum(data, 1)[None]

    def forward(self, x):
        start = time.time()
        with mp.Pool(5) as p:
            result = p.map(self.func_to_apply, x)
        # result = []
        # for x_i in x:
        #     result.append(self.func_to_apply(x_i))
        print(f'Time it takes whole op: {time.time()-start} s')
        x = torch.cat(result, dim=0)
        return self.net2(self.relu(self.net1(x)))


def train():
    print(f"Running Model with parallelized forward pass.")
    model = DemoMultiProcModel()

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(model.parameters(), lr=0.001)

    for batch_idx in range(5):
        # Generate random inputs/labels
        inputs = torch.randn(256, 196, 4)
        labels = torch.randn(256, 5)

        # Train
        optimizer.zero_grad()
        outputs = model(inputs)
        loss_fn(outputs, labels).backward()
        optimizer.step()
        print(f"Batch {batch_idx} done.")


if __name__ == '__main__':
    train()