I think in a output channel, the value of every pixels is the probability that the exact pixel belongs to exact class. So even if in that channel, a block of pixels belongs to the same class, but the values of  the block will be different. So it may be look like noise.
I think you need save the index of the max probability to every pixels, then assign colors to pixel. For example, we can set red color to the pixels which max probability index is 12.