Assuming the encoder is the same one, simply compute forward outputs. If the size of inputs from different datasets is different, conventionally, you need to make them the same size such as cropping them or resizing them to the same size.
I write a brief example for you reference.
enc = model()
fc = nn.Linear(...)
DNN = another_model()

input1 = torch.ones(batch_size, 10)
input2 = torch.zeros(batch_size, 10)

optim = optim.Adam(enc.parameters(), lr=lr)

... some operation for input1
output1 = fc(enc(input1))

... some operation for input2
output2 = DNN(enc(intput2))

loss_1 = cal_loss1(output1, target1)
loss_2 = cal_loss2(output2, target2)

total_loss = loss_1 + loss_2

optim.zero_grad()
total_loss.backward()
optim.step()