Fasterrcnn works seamlessly with both grayscale and rgb. The error is coming somewhere in the forward function where the output from one component is not matching the input to the next component, you will have to debug a little using pytorch hooks to figure it out