Thank you  @JuanFMontesinos so  what I was planning is to creat Net which will be common for 2 inputs,
def forward(self,x):
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

def calculate(self, input1, input2):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2

and calculate the loss between output1 and output2 and based on the difference value, determine the modification