Just as a suggestion, have you tried to use depth_to_space?

Use convolution to increase the depth
Use depth_to_space (combination of view, permute to shuffle) to increase the size

Would that be more stable than interpolate?