Hi! I am recently working on a module like this: the module has 8 different operations like conv3x3, conv5x5, skip_connection, etc. And I know that all the operations produces output in the same shape.
During forward, I want to get the result of each operation, combine them together and get the final result. As I have detected, the for-loop is the bottle-neck of the program. Therefore, I am wondering if there are any methods to make the for-loop more parallel?
An example demonstration of my module is as below.
class Module(nn.module):
    def __init__(self):
        super(Module, self).__init__()
        self.ops = nn.ModuleList()
        for i in range(8):
            # append an operation in self.ops, such as conv3x3
            self.ops.append(...)
    
    def forward(self, x):
        tmp = list()
        for i in range(8):
            tmp.append(self.ops[i](x))
        # func can be any post-preprocessing function, not important
        result = func(tmp)
        return result