well it depends on what you want to plot…
I think you are referring to papers which visualize the network learnable parameters, and the plots aims to make sense in what your model is learning.
In other cases you may only plot the model’s output to understand how it is doing in relate to ground truth / whatever you are trying it will learn.