Well, like @etekiller says, you can try to “Compose Transforms”. An example would be here 319. And then use image to PIL transform 168 to solve your issue. This though will likely add a ton of time overhead to your data loading pipeline!
If you’re not using many transforms from pytorch on your data during “Composing your transform”, why not try using normal tensor operations to go from RGB to Gray? Try the function OpenCV uses like here 149. You can find the reasons for such an averaging here 40