Hello @Andreas_Kopp !
With transfer learning, there are two separate datasets to consider: one used for training the baseline model and another for fine-tuning/training your final model.
In scenarios 1, the privacy of the public dataset is not preserved, but the privacy of the final model is preserved if you train with DP.
In scenario 2, just as in scenario 1, the privacy of the dataset used to train your baseline model is NOT preserved, but only that of the data used to train your final model is. If preserving the data used to train your baseline model is not important for your use case, this might result in better accuracy than that of scenario 1 (depending, ofcourse, on your task as well as the size and distribution of this data compared the public dataset)
In scenario 3, the privacy of all the data is preserved, but the accuracy will be very low.
Hope this adds some clarity.