Hey,
Found the issue. The label encoding was perfectly fine. The mistake I was doing was normalising the images using transforms. This was distorting the values.
Thanks for the support. Really appreciate.