By Resnet-50 do you mean an imagenet pre-trained one? If you have enough data, one option to get some good features would be to train a network (like Resnet or VGG) from scratch using self-supervised approaches like rotation prediction. This will help learn some features specific to your data.