I&#x27;m not sure if this provides any insight, but I think it&#x27;s interesting to visualize this type of thing for the much simpler case of a two-class problem in two dimensions:<p><a href="http:&#x2F;&#x2F;i.imgur.com&#x2F;NdzdH5j.png" rel="nofollow">http:&#x2F;&#x2F;i.imgur.com&#x2F;NdzdH5j.png</a><p>On the left are the training vectors, color-coded by label; the background color-codes the probability output by the learned network at each point in the plane. On the right is the gradient field of the network&#x27;s output corresponding to the blue class. The gradient field shows, at any point, the local direction of greatest increase toward the blue class.