Solved by ptrblck in post #9 
              
              
                Sorry for not being clear enough. 
You could pass both tensors to the forward method and concatenate the activations as seen in this dummy code: 
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv = nn.Conv2d(1, 1, 3, 1, 1)
        self.lin = …