AMP helped a lot here!
It reduced size and inreased performance.
And since I can now use batch_size 8 it really speeds up everything!
Thanks a lot for this advise! 
Still, if anyone has ever succeded in getting AlexNet or VGG to run inside MaskRCNN, please show me how.
Even if it does not help, it would help me to convince my boss that this is not a solution.