Hey, @temi92!
I have applied transfer learning once, and did it that way:

I used a pretrained VGG on Imagenet (your dataset A with weights W1)
Then, I used this VGG to train on Imagenette (a similar dataset, your dataset B)
So in my case, it was best to just continue training with dataset B, however, note that the VGG already converged on the dataset A.

Let me know if you have any questions.
BR, David