Hi, I trying to train auxiliary task using original model’s mid-block features without influence to original model’s performance. I use detach(), however original model’s performance is decreased.
there are two model

the main model
the auxiliary model that takes main model’s mid block

The main network have  Block 1 and fully-connected layer like below,
input -> B1 -> FC1 -> score
then, auxiliary model have only fully-connected layer
input(it will be B1) -> FC2
I want to train moth model simultaneously, auxiliary model doesn’t affect to main model.
here’s my code
two model
MainModel / AuxModel
two optimizer
opM = optimizer(MainModel)
opA = optimizer(AuxModel)
training main model
MainModel .zero_grad()
#when I forward to main_model, I got score and midBlock
mainLoss, B1 = MainModel(input)
mainLoss.backward()
opM.step()
training aux model
AuxModel.zero_grad()
B1 = B1.detach()
prediction= AuxModel(B1)
auxLoss = criterion(prediction)
auxLoss.backward()
opA.step()
I try many things include each model’s param.requires_grad = False, however it still affect to original model’s performance.
could you help me with any advices?