When I convert it into 1 channel (using grayscale) I lose all that color information and get a tensor with zero values and visualizing this image show a totally black image.

Could you show a code snippet that shows us why you get a black image?
As far as I know you have two options:

Convert to grayscale
Adapt the U-Net to accept multiple channels by changing the network architecture.