Solved by ptrblck in post #4 
              
              
                Thanks for the code. 
It looks like to padding of your second max pooling layer is wrong, since you are using the same argument in Keras. 
Try this definition self.maxpool2 = nn.MaxPool2d(7,2,padding=3) and your output will be [batch_size, 96, 4, 4] for both branches.