Ah, you’re right.  @SimonW . I’ve read it briefly and did not notice that @Leaf was trying to do regression. Sorry about that.
Nevertheless, transforming it in a regression is tricky though, since there is no similarity between classes with close-by indices, i.e. class 2 and 3 although close in index space may not depict close-by objects such as road and pavement.  Within the 16 classes there are some very distinct ones such as vehicle, person, road and their indices does not reflect at all their similarity or the distance between them.
This should make your training difficult.