The Building Blocks of Interpretability
If deep learning researchers had a strong theoretical understanding of their
own field, a &quot;grand unified theory of deep learning&quot; (and possibly all
learning) then they wouldn&#x27;t need special tricks to do explanation .<p>Unfortunately, the deep learning field suffers from what John McCarthy used to
call the &quot;Look ma, no hands&quot; approach to AI: that&#x27;s when you get a computer to
do something that hasn&#x27;t beeen done before with a computer and publish a paper
to announce it without any attempt to identify and study any intellectual
mechanisms.<p>So the majority of deep learning papers are result papers: someone tweaks an
existing architecture, or invents a new one, to do something new, or beat the
state-of-the-art results. Theoretical papers are very few and far between and
often come from outside the field (like the Renormalisation Group paper, or
the papers on Information Bottleneck Theory).<p>I don&#x27;t see how work like the one described in the article bucks the trend.
Visualisation may be intuitive, but any two people can see completely
different things in the same image, especially complex images of large scale
activations. The result is an interpretation method that&#x27;s up to, well,
personal interpretation. What&#x27;s more- this only works with vision, where
activations can sort of map to images. It&#x27;s no use for, say, text, sound, or
other types of data (despite what the article says).<p>Articles like this tell me that deep learning researchers have basically given
up on understanding how their own stuff works in the course of their careers
and simply accept that their success must come from beating benchmarks and
producing pretty graphs.<p>A pitty.