Perhaps you can try freezing some layers, and only fine-tune a specific layer.
In PyTorch, this can be done as follows:
for name, param in model.named_parameters():
     if name == ...:
        param.requires_grad == False