I try to train multiple models within each batch, since the loss functions are based on the result of all models. Each model has their own dataset and optimizer. However, when I try to do backward() on the second model, “modified by an inplace operation” error occurs. I do set detect_anomaly to True, but the returned traceback makes no sense to me. Here is the code:
    for epoch in range(num_epochs):
        for i in range(len(splitted_v1)):
            X1_ = model_v1(splitted_v1[i].float())
            S1 = similarity_matrix(X1_)

            X2_ = model_v2(splitted_v2[i].float())
            S2 = similarity_matrix(X2_)

            local_s = (S1 + S2) / 2

            loss_v1 = custom_loss(X1_, splitted_v1[i], S1, local_s)
            optimizer_v1.zero_grad()
            loss_v1.backward(retain_graph=True)
            optimizer_v1.step()

            loss_v2 = custom_loss(X2_, splitted_v2[i], S2, local_s)
            optimizer_v2.zero_grad()
            loss_v2.backward(retain_graph=True)
            optimizer_v2.step()

Here is the traceback:

Blockquote
Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 336, in 
X1_ = model_v1(splitted_v1[i].float())
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 102, in forward
x = self.decoder(x)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 100, in forward
input = module(input)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 87, in forward
return F.linear(input, self.weight, self.bias)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
(print_stack at /Users/distiller/project/conda/conda-bld/pytorch_1587428061935/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 351, in 
loss_v2.backward(retain_graph=True)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/autograd/init.py”, line 100, in backward
allow_unreachable=True)  # allow_unreachable flag
Blockquote

How the call on model_v1 influences the variable of model_v2.
I am new to pytorch. Any help would be appreciated!