Ok I was dumb. Will keep the post and answer, in case someone like me appears…
The issue was with following lines :
X = (np.transpose(X, (1, 2, 0)) + 1) / 2.0 * 255.0     # 15.9999
X= X.astype(np.uint8)  # Truncates the 0.9999 part..

I should have rounded X before converting to uint8 dtype
X = (np.transpose(X, (1, 2, 0)) + 1) / 2.0 * 255.0     # 15.9999
X= np.rint(X)
X= X.astype(np.uint8)  # Truncates the 0.9999 part..