How did you measure the time using RGB and gray images?
Are you using multiple workers for the preprocessing?
I’m not sure you’ll see a huge performance gain using grayscale images.
Which error do you get if you convert all images?