What I’m trying to do in the training loop is use the output of one model as the the ground truth label of the other.
Training loop -
for e in range(epochs):

  epoch_loss_t = 0
  epoch_loss_s = 0

  for i, (data, label) in enumerate(trainloader):

    data, label = data.to(device), label.to(device)

    t_out = teacher(data)
    t_loss = F.cross_entropy(t_out, label)
    t_loss.backward()
    t_optim.step()
    t_optim.zero_grad()
    epoch_loss_t += t_loss.item()

    s_out = student(data)
    s_loss = (1 - alpha) * F.cross_entropy(s_out, label)
    s_loss += alpha * loss_fn(s_out, t_out)
    s_loss.backward()
    s_optim.step()
    s_optim.zero_grad()
    epoch_loss_s += s_loss.item()


Is there a way I can somehow detach the output of the the first model so that a separate computation graph is created for the second one ? I’m guessing that simply doing a .detach() on the output of the first model should work.