Yes, I tried it and resnet18 works fine 
And I will try the mixed precision, if I find some howto…
I still would love to experiment with very small or even self made backbones.
Its not only that my use case has 50 instead of 1000 classes, its also that I have very few samples only.
So it is essiential that I reduce the number of weights/parameters to a minimum, to have a fast learning  model.
I have tried the 50 classes without detection (just as cutout images) and could train them with AlexNet or VGG10 with 15 samples each class (and a lot of image augmentation) with 80-85% accuracy. And now I would like to do the same with the Mask RCNN around it to detect the objects first.
I had the impression that the backbone can easily be exchanged, but it seems the adaptation is very specific and not obvious. Therefore I need some help here.