neerajshukla1911:

x = torch.cat([x1.view(-1, 4), x2.view(-1, 4), x3.view(-1, 4)], -2)


Not quite answering the question you have but this line above replaces the output of the first convolution layer:
x = self.conv1(x)
x = self.act1(x)
x = self.pool1(x)

Is this a desired approach?