Found on this forum:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
        )

    def forward(self, x, y):
        x1 = self.features(x)
        x2 = self.features(y)
        x = torch.cat((x1, x2), 1)
        return x

By using x = torch.cat((x1, x2), 1) one can concatenate multiple towers/blocks. The forward function can take multiple input data. Want to learn more, search for “Pytorch multi input network” or something similar.