Solved by ptrblck in post #2 
              
              
                I’m currently not seeing, where the batch sizes 32 and 128 are coming from, but I guess the torch.cat call yields the error, since it concatenates the tensors in dim0 by default. 
Try to use out = torch.cat((x1, x2, x3, x4), 1) instead.