It seems the image itself contains these additional values and you can check the first 16 unique values via:
img = PIL.Image.open("./color_test.png")
a = np.array(img)
u = np.unique(a)

f, axarr = plt.subplots(4, 4)
for idx, ax in enumerate(axarr.reshape(-1)):
    if idx == 16:
        break
    ax.imshow(a == u[idx], cmap='gray')

Which shows the expected blobs as well as the (unwanted) borders:

so you would have to make sure to either filter them after loading the image or during the creation of it.