My guess is that the output might not be a clear interpolation between the two original classes, since your model got a new and unknown input.
Similar behavior can be seen, if you train a GAN with a latent vector of dim Z by sampling from a Gaussian distribution. If you try to directly interpolate between two valid points, your input would leave the “dense” part of the distribution (i.e. the high-dimensional bubble) and your model could output artifacts.
PS: let us know what the outcome is, in case you are running these experiments!