Yes that info is lost in the tensor. If you want the original colors used you have to save the palette.
If you are ok with the red being for example green and the yellow being blue, you can use a new palette.