You might want to try the Pandas function instead. You mentioned using a DataFrame in your original post so I figured you were using Pandas.
You can also set up a list of class names like ['red', 'green', 'blue'] and use the index (0 = red, 1 = green, etc) as categories for the to_categorical function. Since the categorical column is your label, you could also try using sparse_categorical_crossentropy as your loss function and use the index values as your labels to skip the to_categorical function altogether.