Oh, if you used cv2 in the training, you might have also trained with BGR (e.g. the original detectron was, I think). I had understood your reply as that you used the usual TorchVision pipeline which uses RGB images. I should have asked better, sorry.
In general, the TorchVision models work with RGB, so most pretrained models do.