Take two vgg net and load the pretrained weight. Write a dataloader which gives you a pair of images one for task1 and other for task2. Pass them to the network and fine-tune them using loss.backward ()