@ptrblck, this way I am able to visualize what all patterns are learnt by base neural networks, but their output is passed in the form of 10 scores, which is then concatenated in higher level neural network, how do I visualize what  are the higher level neural networks learning, currently I do something like this, I add image output from base networks, and plot them
def forward(self, x):
    img = self.modelone(x)[0] + self.modeltwo(x)[0]
    grid = torchvision.utils.make_grid(img)
    writer.add_image('img_combine', grid)
    img_2d = self.encoder(img)
    img = self.lin_two(self.lin_one(img_2d.view(img_2d.size(0), -1)))
    out = torch.cat((self.modelone(x)[1], self.modeltwo(x)[1], img), dim=-1)
    out = (self.linear((out)))
    return img_2d, out

where ModelOne forward looks like
def forward(self, x):
    x = self.encoder(x)
    return (x, self.lin_two(self.lin_one(x.view(x.size(0), -1))))

encoder is conv, batchnorm, relu