You are probably adding more and more onto the autograd forward graph here … If you don’t want to backpropagate from x3 -> x1, add
with torch.no_grad():
    x1 = self.cnn(x1)
    x2 = self.cnn(x2)
    x3 = self.cnn(x3)