So in more detail, I’d say

do what has worked well for experienced people - this is what using vgg-like is. Many people advocate using the weights as well (ie do finetuning)
In his excellent fast.ai 127 courses, Jeremy Howard recommends using a lot of capacity to overfit and then reduce from there. I can wholeheartedly recommend the course
Experiment, experiment, experiment. People have built automatic architecture searches, that probably is because you don’t know before you tried.

Best regards
Thomas