antoon:

Maybe should I import the weights of the imported network in some way to make it work?


You should use the pretrained model or load a state_dict for an already trained model, if you want to fine tune it.
Which parameters are you freezing and is the training as well as validation accuracy low or just one of them?