summaries: This paper proposes to empirically investigate to what extent large language models like GPT-3 learn (or "distill") a simplicity prior, i.e., a prior that favors concepts with shorter description length. The authors investigate this by randomly choosing 8 concepts/programs in the P3 language whose total number of P3 instructions number from 1 to 8, respectively. Using the framework of Telle et al. (2019), the authors are able to determine the "witness set," the minimal number of examples necessary for a learner with the correct simplicity prior to distinguish the intended concept from others. This witness set is fed to a learner, which is then evaluated on 5 randomly selected test examples of the concept. The authors also consider feeding additional sets of 2 and then 3 examples to the learner, each with its own random test set. The authors evaluate the accuracy of GPT learners, human learners, and 2 inductive program learners on these 3 test sets. They also elicit explanations from the learners.

summaries: This paper empirically compares humans, language models, and inductive programming systems through the framework of machine teaching. Concepts and example sets are selected using a simplicity prior with reference to the P3 language. The main experiments present the example sets to the humans or machines and ask them to infer the corresponding concept. This setup leads to several experimental questions regarding the alignment between humans and the computational approaches and the extent to which language models like GPT have distilled a simplicity prior.

summaries: This paper studies the type of patterns learned by large pre-trained language models (LMs) in a few-show setting. The authors differentiate between external patterns (e.g., common-sense or world-knowledge), and internal patterns, such as ABAB, and focus on the latter. They frame this as a teaching problem with strong priors, and study the size of the example set (number of shots in the few-shot setting) compared to the length of explanation that can be provided to the model in order to teach it the given pattern. The authors focus on the simplicity prior, and study whether it is implicitly encoded in such large models. Experiment that compare LMs, program induction systems and humans are performed. The main findings are that LMs perform as well as humans in few-shot settings, but the two populations differ in their ability to generate explanations.  I would say upfront that I am not an expert in this topic, and have found many points in the paper unclear, perhaps due to this. However, I wasn't fully convinced by the setup used by the authors. I am looking forward to the author response that might clarify some of these points.

summaries: This paper attempts to ask whether language models have a bias that allows them to learn from fewer examples in cases where a rule is simpler. They compare exploitation of rule simplicity in a variety of models, including human participants. This prior towards simplicity is most strongly observed in larger models, but the minimal example set is rarely enough to train any system, so with the exploitation of simplicity is not a perfect prior.  The paper also provides an analysis of the capability of these systems to provide explanations of the rule, but only humans and the algorithm that they specifically select for this purpose were capable of providing such explanations.

summaries: In this paper, the authors compare the few-shot learning abilities of humans and GPT* language models (and inductive programming systems) from a machine teaching point of view. Models (and humans) are given a minimal witness set (examples using as few bits of information as possible) that should be sufficient to uniquely determine a program (function from a sequence of bits to another) under a pretty strict simplicity prior. The authors also examine whether additional examples (on top of the minimal witness set) help performance. They consider 8 different P3 programs of increasing complexity based on the number of operators. Examples are presented as English prompts.  The minimal witness set is often not sufficient for humans and LMs to generalize correctly. Nevertheless, there is some evidence that GPT models favor simplicity to some degree. Overall, larger GPT models generally perform better than smaller ones, but the correlation is not perfect. GPT-3 models often generalize better than the human candidates based on the few provided examples. However, humans sometimes provide acceptable explanations (for the shorter programs), but the authors could not get GPT models to explain their reasoning.

abstractText: Large language models have recently shown a remarkable ability for few-shot learning, including patterns of algorithmic nature. However, it is still an open question to determine what kind of patterns these models can capture and how many examples they need in their prompts. We frame this question as a teaching problem with strong priors, and study whether language models can identify simple algorithmic concepts from small witness sets. In particular, we explore how several GPT architectures, program induction systems and humans perform in terms of the complexity of the concept and the number of additional examples, and how much their behaviour differs. This first joint analysis of language models and machine teaching can address key questions for artificial intelligence and machine learning, such as whether some strong priors, and Occam’s razor in particular, can be distilled from data, making learning from a few examples possible.

authors:
L. Ai
S.H. Muggleton
C. Hocquette
M. Gromowski
U. Schmid

title: Beneficial and harmful explanatory machine learning

venue: Machine Learning,

authors:
S. Basu
J. Christensen

title: Teaching classification boundaries to humans

venue: In AAAI

authors:
E.M. Bender
T. Gebru
A. McMillan-Major
S. Shmitchell

title: On the dangers of stochastic parrots: Can language models be too big

venue: In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency,

authors:
E.M. Bender
A. Koller

title: Climbing towards NLU: On meaning, form, and understanding in the age of data

venue: In ACL,

title: On a family of Turing machines and the related programming language

venue: ICC Bulletin,

authors:
T.B. Brown
B. Mann
N Ryder

title: Language models are few-shot learners

venue: arXiv preprint arXiv:2005.14165

authors:
M. Cakmak
A.L. Thomaz

title: Eliciting good teaching from humans for machine learners

venue: Artificial Intelligence,

authors:
N. Chater
P. Vitányi

title: Simplicity: A unifying principle in cognitive science

venue: Trends in Cognitive Sciences,

authors:
J. Degen
R.D. Hawkins
C. Graf
E. Kreiss
N.D. Goodman

title: When redundancy is useful: A bayesian approach to “overinformative

authors:
J. Devlin
Chang
M.-W
K. Lee
K. Toutanova

title: Bert: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805

authors:
K. Dingle
C.Q. Camargo
A.A. Louis

title: Input–output maps are strongly biased towards simple outputs

venue: Nature communications,

authors:
D.L. Dowe
J. Hernández-Orallo
P.K. Das

title: Compression and intelligence: social environments and communication

venue: In International Conference on Artificial General Intelligence,

authors:
M. Garcia-Piqueras
J. Hernández-Orallo

title: Optimal teaching curricula with compositional simplicity priors

venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,

authors:
N.D. Goodman
M.C. Frank

title: Pragmatic language interpretation as probabilistic inference

authors:
J. Gou
B. Yu
S.J. Maybank
D. Tao

title: Knowledge distillation: A survey

venue: International Journal of Computer Vision,

authors:
S. Gulwani
J. Hernández-Orallo
E. Kitzelmann
S.H. Muggleton
U. Schmid
B. Zorn

title: Inductive programming meets the real world

venue: Communications of the ACM,

authors:
D. Hendrycks
C. Burns
S. Basart
A. Zou
M. Mazeika
D. Song
J. Steinhardt

title: Measuring massive multitask language understanding. ICLR

authors:
J. Hernández-Orallo
C. Ferri

title: Teaching and explanations: aligning priors between machines and humans

venue: Human-Like Machine Intelligence,

authors:
J. Hernández-Orallo
J.A. Telle

title: Finite and confident teaching in expectation: Sampling from infinite concept classes

venue: In ECAI,

authors:
D. Hupkes
V. Dankers
M. Mul
E. Bruni

title: Compositionality decomposed: How do neural networks generalise

venue: Journal of Artificial Intelligence Research,

authors:
G. Izacard
E. Grave

title: Leveraging passage retrieval with generative models for open domain question answering

venue: arXiv preprint arXiv:2007.01282

title: Magichaskeller: System demonstration

venue: AAIP

authors:
F. Khan
B. Mutlu
J. Zhu

title: How do humans teach: On curriculum learning and teaching dimension

venue: In NIPS,

authors:
Y. Kirstain
P. Lewis
S. Riedel
O. Levy

title: A few more examples may be worth billions of parameters

venue: arXiv preprint arXiv:2110.04374

authors:
N. Kühl
M. Goutier
L. Baier
C. Wolff
D. Martin

title: Human vs. supervised machine learning: Who learns patterns faster? arXiv preprint arXiv:2012.03661

authors:
Y. Lakretz
G Kruszewski

title: The emergence of number and syntax units in LSTM language models

venue: NAACL

authors:
M. Li
P. Vitányi

title: An introduction to Kolmogorov complexity and its applications (3rd ed.)

authors:
P. Liu
W. Yuan
J. Fu
Z. Jiang
H. Hayashi
G. Neubig

title: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

venue: arXiv preprint arXiv:2107.13586

authors:
Y. Lu
M. Bartolo
A. Moore
S. Riedel
P. Stenetorp

title: Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

venue: arXiv preprint arXiv:2104.08786

authors:
G. Marcus
E. Davis

title: GPT-3, bloviator: OpenAI’s language generator has no idea what it’s talking about

venue: Technology Review

authors:
F.S. Melo
C. Guerra
M. Lopes

title: Interactive optimal teaching with unknown learners

venue: In 27th International Joint Conference on Artificial Intelligence,

authors:
S. Patsantzis
S. Muggleton

title: Top program construction for polynomial meta-interpretive learning

venue: Machine Learning

authors:
A. Radford
J. Wu
R. Child
D. Luan
D. Amodei
I. Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI blog,

title: A universal prior for integers and estimation by minimum description length

venue: Annals of Statistics,

authors:
H. Shah
K. Tamuly
A. Raghunathan
P. Jain
P. Netrapalli

title: The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07710

title: Communication theory of secrecy systems

venue: The Bell system technical journal,

authors:
K. Sinha
S. Sodhani
J. Dong
J. Pineau
W.L. Hamilton

title: CLUTRR: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv:1908.06177

title: A formal theory of inductive inference

venue: Part I. Information and Control,

authors:
W. Sun
O. Nasraoui
P. Shafto

title: Evolution and impact of bias in human and machine learning algorithm interaction

venue: Plos one,

authors:
J.A. Telle
J. Hernández-Orallo
C. Ferri

title: The teaching size: computable teachers and learners for universal languages

authors:
S. Toshniwal
S. Wiseman
K. Livescu
K. Gimpel

title: Learning chess blindfolded. https://openreview.net/pdf?id=DGIXvEAJVd

authors:
G. Valle-Perez
C.Q. Camargo
A.A. Louis

title: Deep learning generalizes because the parameter-function map is biased towards simple functions. ICLR

authors:
A. Vaswani
N Shazeer

title: Attention is all you need

venue: In Advances in neural information processing systems,

authors:
C.S. Wallace
D.M. Boulton

title: An information measure for classification

venue: Computer Journal,

authors:
A. Webson
E. Pavlick

title: Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247

title: The lack of a priori distinctions between learning algorithms

venue: Neural computation,

authors:
S. Xu
S.J. Semnani
G. Campagna
M.S. Lam

title: AutoQA: From databases to QA semantic parsers with only synthetic training data

authors:
Yang
S.C.-H
P. Shafto

title: Explainable artificial intelligence via Bayesian teaching

venue: In Workshop on Teaching Machines,

authors:
T.Z. Zhao
E. Wallace
S. Feng
D. Klein
S. Singh

title: Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690

title: Machine teaching: An inverse problem to machine learning and an approach toward optimal education

venue: In AAAI,

heading: 1 Introduction

text:
An effective and reliable interaction between humans and machines depends on a proper understanding of their priors. The alignment between these priors, in the form of knowledge and other predispositions used as bias, is especially critical for communication and learning (Sun et al., 2020). For instance, after a human utters the prompt “on off on off on”, another human would expect “off” to be a more likely continuation than “on”. In contrast, after the prompt “twinkle twinkle little” we would expect “star” to be more likely. The former depends on an abstract pattern intrinsic to the sequence (a simple algorithmic sequence alternating “on” and “off”), while the latter depends on some language pattern extrinsic to the sequence (common use in the English language because of a popular song).
Language models are a kind of AI systems extrapolating sequences of symbols, and are very good at exploiting the extrinsic patterns that are borrowed from humans after training over massive natural language corpora. However, it is only recently, with systems such as BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), that we see how these models can also discover some intrinsic patterns provided in the prompts. This kind of inference is called ‘few-shot learning’, even if there is no further training or fine-tuning of the model. Many of these language models are based on transformers and other attention-based architectures (Vaswani et al., 2017), which do not feature recursion or loops, unlike some approaches based on recurrent neural networks. Some particularly well-chosen and optimised prompts have shown remarkable extrapolations (Xu et al., 2020; Izacard and Grave, 2020; Hendrycks et al., 2021), illustrating this new kind of abstraction on the go. However, it is unclear how elaborate or algorithmic these abstractions can be. More studies are needed to determine the optimal prompts and example order that make the model capture
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
a range of different patterns (Zhao et al., 2021; Lu et al., 2021; Liu et al., 2021; Webson and Pavlick, 2021; Kirstain et al., 2021). Even when the prompt template is fixed, it is an open question what the minimal set of examples should be for learning a concept, how this minimality is evaluated and what priors pretrained models incarnate for few-shot learning to work.
Precisely, machine teaching is the area of artificial intelligence that looks for the optimal examples or curriculum that a teacher should use to make a learner capture a concept (Zhu, 2015). This is of enormous importance today in applications such as digital assistants, where we would like to teach them new concepts or routines with a few examples, in the same way humans teach or communicate with other humans (Cakmak and Thomaz, 2014; Goodman and Frank, 2016; Degen et al., 2020). It is also fundamental for explainable AI, as the dual problem appears: how a machine can teach (explain) a concept to a human in an efficient way (Khan et al., 2011; Basu and Christensen, 2013; Hernández-Orallo and Ferri, 2021). When dealing with rich representation languages (e.g., regular languages or Turing-complete languages), closer or similar to human language in expressive power, it has recently been shown that strong simplicity priors can make effective teaching possible with very few examples: the overall size of the example set is usually shorter than the algorithmic expression of the concept itself (Telle et al., 2019). For instance, teaching the concept of what an odd number is becomes shorter by using a few examples than by giving the full definition of what an odd number is.
Language models and machine teaching can now meet because of this new few-shot learning ability of language models. We can explore the question of how many examples are needed (or how long they have to be) to teach something to a language model. These examples and their template are seen as teaching prompts: we want the language model to continue the sequence as if it had captured the pattern that has been expressed in the prompt. From the machine teaching perspective, language models are general systems, as they have not been trained for simplicity or for any particular domain, other than a partial sample of human knowledge expressed as natural language.
With their conjunction, we can now ask key questions for AI such as the relevance of simplicity or acquired priors to make learning from a few examples possible (Chater and Vitányi, 2003). In this paper, we examine how a machine teaching setting inspired by Occam’s razor can help us understand how language models ‘learn’ and choose prompts more efficiently. We use the term ‘prior distillation’, overloading the notion of knowledge distillation within the field of machine learning, as we analyse whether a simplicity prior is acquired as the result of training large language models such as GPT-2 or GPT-3. To complete the picture, we also compare them with some other AI systems (e.g., program induction systems) and humans on the same task, which are known to have this prior.
The key contributions of this paper are:
• We analyse language models in the context of machine teaching, using minimal teaching sets for few-shot inference.
• We compare the acquired priors of language models with humans and program induction systems (Gulwani et al., 2015), when facing patterns preferred by a simplicity prior.
• We find that language models such as GPT-3 have similar or better performance than humans. The performance negatively correlates with the complexity of the concept.
• Language models and humans differ in their capability for producing explanations. The length of human explanations correlates with the complexity of the concept.
Having a better understanding of language models and its relation to the way other systems, including humans, make inferences is fundamental research that can make the use of these systems safer and more effective, through more robust prompts to solve a variety of tasks. However, this better understanding can also be used maliciously to cunningly modify prompts that alter the results of some applications built on top of language models. A query may fail when using dates, numbers or other strings that have intrinsic patterns. For instance, GPT-3 usually continues “Mary’s father has 5 daughters, Lucy, Anne, Alice, Jane and” correctly, but fails on “Mary’s father has 5 daughters, Ba, Baba, Bababa, Babababa and”, systematically continued with ‘Bababababa’.
The rest of the paper follows with related work in language models, the comparison of machine and human learning, as well as Occam’s razor and machine teaching. Section 3 contains the experimental design, and introduces the systems we consider for comparison. Section 4 presents and analyses the results. Finally, we close the paper with a discussion of the implications of this work. The supplementary material includes additional experiments, plots, tables and specific analyses.

heading: 2 Related Work

text:
It is only recently that deep language models are starting to show some minimal capabilities for few-shot learning. Systems such as GPT-3 solve “tasks which require to perform simple on-the-fly computational reasoning, recognize a novel pattern that is unlikely to have occurred in training, or adapt quickly to an unusual task” (Brown et al., 2020). Examples of these tasks are simple arithmetic operations and the induction of symbolic manipulations such as anagrams or reversed words. Hendrycks et al. (2021) compare the performance of a text model with humans on 57 tasks including elementary mathematics, US history, law, and more. Sinha et al. (2019) show that language models obtain poor robustness and generalisation capabilities when compared to graph neural networks that work with symbolic inputs. The experiments also show that humans (with limited time) performed well in simple problems but the performance decreases with complex examples. There are also important limitations of language models on simple generalisation tasks (Bender and Koller, 2020; Marcus and Davis, 2020). On the one hand, recurrent neural networks, such as long-short-term-memory (LSTM) architectures, have shown the capacity of capturing syntax-sensitive generalisations such as long distance number agreement (Lakretz et al., 2019), or language modelling for the game of chess (Toshniwal et al., 2021). The abstract reasoning capacity of transformers and other deep learning architectures has been studied by Hupkes et al. (2020). In sum, as language models based on transformers are not recursive, it is an open debate whether they are just memorising a huge corpus, or they really have the capacity of learning abstract concepts from data.
The comparison between humans and AI systems has usually been performed with systems that are trained for a task, but not for a variety of tasks that are introduced with a few words or examples. For instance, Kühl et al. (2020) present an experiment that compares the performance of several machine learning algorithms and 44 human participants recognising patterns in 3x3 matrices. The results in this case show that humans, as expected, tend to achieve better performance than the AI systems when few examples are provided. This work suggests that many AI systems may not be competitive with humans when limited examples are given for a range of tasks. Further insight can be obtained when we try to look into the extracted models through explanations, and analyse them in the context of concept complexity. For instance, the relation between complexity of concepts and explanations is discussed by Ai et al. (2021).

heading: 2.1 Occam’s Razor

text:
Occam’s razor, the a priori preference for simple hypotheses over more complex ones, has pervaded philosophy of science, inductive inference, and human communication, and it is ubiquitous in statistics and machine learning, explicitly —through criteria such as the MML/MDL principles (Wallace and Boulton, 1968; Rissanen, 1983) and the use of regularisation terms— or implicitly —in the optimisation algorithms or parameter-function maps in many methods, including deep learning (Valle-Perez et al., 2019; Shah et al., 2020). Occam’s razor is backed by strong theoretical and empirical reasons (Solomonoff, 1964; Dingle et al., 2018), which oppose the idea that all hypotheses should be equally likely a priori, an assumption that would lead to the no free lunch theorems results (Wolpert, 1996), making learning impossible. While simplicity is always relative to a reference language, the invariance theorem (see, e.g., Li and Vitányi, 2008) ensures that the use of different (optimal) languages gives at most a constant difference that only depends on these two languages. While these constants may be very large, some level of agreement in what simplicity is can happen even for very different representations, as we explore in this paper.
A prior for simplicity is hard to understand in large language models, a kind of architecture that is usually overparametrised, compared to the size of the evidence. While the model is not necessarily the most compressed representation of the data, it leads to continuations with low perplexity, which can be used to compress future data, as language models were introduced for communication (Shannon, 1949). It is this interpretation that sheds light on the role of priors, compression and simplicity in human communication and understanding (Chater and Vitányi, 2003; Dowe et al., 2011). It is precisely in the context of few-shot inference, once the model has been trained, that we can reformulate the question as whether these models make inferences based on a preference for simpler hypotheses. We use the term ‘distillation’, in a somewhat related way to ‘knowledge distillation’ (Gou et al., 2021), to represent that the simplicity prior has emerged or have been distilled in the trained model, and used as ‘learning’ bias in few-shot inference. For further discussion about Occam’s razor, compression and inference, see Section 15 in the supplementary material.

heading: 2.2 Machine Teaching and Priors

text:
Machine teaching has analysed learning from a minimal set of examples for different concept classes, with the representational power of the class being a major factor to determine when this is possible. Machine teaching has also been used to understand how humans teach machines, comparing a machine teaching setting with a curriculum learning setting (Khan et al., 2011; Basu and Christensen, 2013). Many of these studies use feature vector concepts instead of rich representations (Turingcomplete) such as natural language. Only recently has machine teaching focused on universal languages, and how relevant simplicity priors become when teaching concepts with very few —or very short— examples (Telle et al., 2019; Hernández-Orallo and Telle, 2020; Garcia-Piqueras and Hernández-Orallo, 2021; Hernández-Orallo and Ferri, 2021).
More specifically, machine teaching is often understood as an inverse problem to machine learning (Zhu, 2015), discovering the best set of examples or curricula for a class of concepts and learners. Most of the early research in machine teaching was based on the idea of the teaching dimension, the minimum number of examples the learner needs to identify a concept. However, the teaching dimension approach is too narrow. For instance, in the case of teaching concepts from richer languages, it has been demonstrated that even if the teaching dimension is small (low cardinality), the examples might be enormously large. Telle et al. (2019) propose a machine teaching framework that is based on the size of examples instead of the dimension of the witness set. While the teaching dimension is related to algorithmic learning theory, the teaching size is more appropriate for language models and constructive examples. For instance, considering two witness sets of the same teaching dimension, the input-output example (010101, 1) contains more information than (0, 1).
Examples are defined as pairs (i, o) of elements (e.g., binary strings) and concepts as functions mapping inputs to outputs. An example set S = {(i1, o1), . . . , (ik, ok)} is just a finite set of inputoutput pairs, used as witness for the teaching process. Considering a representational language L for programs, a learner Φ can be seen as a function mapping sets of examples to programs. A simplicity-based learner on L just chooses the shortest L program that is compatible with the witness set. As a result, a witness set only needs to distinguish the shortest satisfying program from earlier, shorter ones. Finally, if a teacher wants to teach a concept c it will look for the shortest witness set S (in bit size) such that Φ(S) returns a program in L that is compatible with c. For a more formal definition of the teaching size, the reader is referred to section 14 in the supplementary material.
In this work, we build on the increasing relevance of the alignment of priors between learner and teacher (Goodman and Frank, 2016; Yang and Shafto, 2017; Melo et al., 2018) to investigate how long witness sets have to be for different learners (language models, program induction systems and humans), when devising the minimal teaching example set using simplicity (Occam’s razor) as the learner’s prior. This is a well-founded controlled experiment setting to analyse the alignment of priors (learning bias) between the source of the concepts (what we call the teacher) and the learner.

heading: 3 Experimental Setting

text:
The following questions will be explored experimentally (and on average terms):
• Q1 - Abstraction capability: Is a language model such as GPT able to capture algorithmic patterns from very few examples selected with a simplicity prior?
• Q2 - Learning curves: Does accuracy increase from the theoretically minimal witness sets needed for teaching the concept with additional examples?
• Q3 - Characteristic curves: How do language models, program induction approaches and humans behave in terms of the complexity of the patterns?
• Q4 - Similarity: How similar are the errors of language models compared to humans and program induction approaches?
• Q5 - Architecture: Apart from being more capable, which has already been shown before, do larger models have a stronger preference for simple concepts than smaller models?
• Q6 - Explanations: Can we get explanations from GPT for these algorithmic patterns? Is the accuracy of human explanations related to performance, and their size related to complexity?
We now present the language, the machine teaching algorithm and the learners.

heading: 3.1 The Machine Teaching Setting

text:
The language L we will use to express programs will be the simple but Turing-complete programming language P3, a variation of P'' (Böhm, 1964) with only 7 instructions. We employ P3 as a concept language because it is very different from the representational languages used by the learners we will use (natural language for humans and functional or logical languages for program induction), and especially different from the internal representation of a language model (transformers). This is also a way to see the invariance theorem in action, as all representational languages are different.
P3 programs operate on an input tape and a write-only output tape, with cells containing one of the three characters in the alphabet: Σ = {‘0’, ‘1’, ‘.’}. The pointer is moved using the instructions or operators ‘<’ (left) and ‘>’ (right). The instructions ‘+’ and ‘−’ are used to respectively increase or decrease the value of the pointed cell, in the (cyclic) order they are defined in Σ. The instructions ‘[’ and ‘]’ represent the start and the end of a loop respectively, exiting the loop when the pointer is on ‘.’. Finally, the instruction ‘o’ is used to output the value of a pointed cell.
We follow Telle et al. (2019) to generate ‘the teaching book’: all P3 concepts and the associated witness sets (assuming a simplicity-based learner). The examples are restricted to binary strings, with ‘.’ used as delimiter, padding cells before and after the string. We use a maximum teaching size of 7 (total number of bits in all the inputs and outputs in the witness set). Removing repeated sets and contradictory examples, the example set space is hence comprised of 17,252 example sets.

heading: 3.2 Choice of Concepts and Examples

text:
We extract eight different concepts c1, . . . , c8, each ci being randomly selected from all programs in the teaching book with i instructions. We bin the concepts into four levels: the programs with 1-2 operators (very low complexity, VL-C), those with 3-4 operators (low complexity, L-C), those with 5-6 operators (high complexity, H-C) and those with 7-8 operators (very high complexity, VH-C).
The minimal witness sets for these concepts are all between 1 and 3 examples. This will be the first batch (the witness set, WS) used as training set. This would lead to perfect identification if the learner were using the same simplicity prior over the same language. But, as the learner’s prior might differ from the teacher’s, we also want to analyse how learning improves as more examples are provided. Therefore, we will use 5 additional examples in two different batches, a first batch with 2 extra examples (additional set one, AS I) and a second batch with 3 extra examples (additional set two, AS II). These two additional sets cannot follow the machine teaching protocol (as they are redundant for perfect alignment). Instead of measuring their size, but to keep it in a similar distribution as the rest, we extract new examples from the example sets for the same program in the teaching book. For the test examples, we generate 5 new examples per concept, using random sampling for the inputs i (avoiding repeated strings in WS, AS I and AS II) with probability 2−(l(i)+1). In total, we have 8 × 3 × 5 = 120 predictions. This number of concepts, batches and test examples aims to find a trade-off for a minimally robust evaluation and the limitations of conducting the experiments with humans.
Table 1 shows two concepts, c1 and c4, their code in P3, an intuitive description, and the several sets (WS, AS I, AS II and Test). The Tables 1, 2, 3 and 4 in the supplementary material contain all used concepts. With all this, the proposed teaching protocol for each ‘concept lesson’ is divided into three phases: (1) we ask to predict the test examples after only seeing WS, (2) we ask to predict the same 5 test examples after seeing WS + AS I and (3) we ask to predict the same 5 test examples after seeing WS + AS I + AS II.

heading: 3.3 Learners

text:
The following learner families have been considered.
Humans: 30 volunteering (unpaid) human participants with ages between 18 and 54. Apart from the answers to each test example, humans had to write an explanation in natural language for each of the 8 concepts after the third phase. All data collected in the experiments were anonymised, informing the participants of the data treatment and their rights, following data protection laws. The form can be found in the GitHub repository (link at the end of the next section).
MagicHaskeller (MH): A general-purpose inductive functional programming system based on systematic exhaustive search (Katayama, 2011). It infers programs expressed in Haskell from inputoutput examples. A strong simplicity prior and a library of primitives allow MH to solve many problems using only one example. When MH returns more than one hypothesis, we take the first one.
Louise: An inductive logic programming system based on Top program construction (Patsantzis and Muggleton, 2021). It learns Prolog programs from finite examples, with the capability of learning recursive programs in polynomial time. Two different approaches are considered: Louise-ens, where the most frequent result returned by the totality of the Top program will be used as the final answer, with ties are broken randomly, and Louise-exp, where the expected score is calculated considering all the possible solutions returned by the Top program.
GPT-2: A transformer-based language model trained on a dataset of 8 million web pages (Radford et al., 2019). The model returns the next predicted word or token considering the previous tokens as the given context. We use the 774M-parameter version of GPT-2. The default hyperparameters of the model are preserved, with the exception of the ‘length’ of the output returned, which is modified to return 3 tokens. By doing so, the experiment execution performance is improved, without asking GPT-2 for more information than what we require for our analysis. Deterministic results can be obtained by setting the ‘top-k’ parameter to 1, but we can get better performance if we use several continuations. Accordingly we will use the default ‘top-k’ value, which is 0. With this, two different approaches will be considered by running the model extrapolations n=40 times for the same query: GPT-2-ens, where the most frequent result (out of the n results obtained for the same query) will be used as the final answer, with ties are broken randomly (this approach can be understood as a majority ensemble setting), and GPT-2-exp, where the n results for the same query are averaged (this approach can be understood as the expected value for GPT-2).
GPT-3: Another series from the GPT family (Brown et al., 2020). We used different available models, with an increasing number of parameters (Ada, Babbage, Curie and Da Vinci), as recent work (Bender et al., 2021) demonstrates that larger models outperform smaller ones. We want to show whether this trend holds for abstract concepts too, with Occam’s razor emerging more strongly for larger models. We will employ acronyms GPT-3A, GPT-3B, GPT-3C, GPT-3D correspondingly. We performed the experiment considering two different settings: the hyperparameter temperature set to 0 and to 1. When the temperature is set to 1, the results are non-deterministic, and the same approach used in GPT-2-ens is applied. We will use T0 or T1 as shortcuts depending on the value of this parameter. For instance, GPT-3 Da Vinci with temperature 0 will be denoted by GPT-3D-T0. Finally, we also asked GPT-3 to provide an explanation of the concepts, just after each batch (8 × 3 explanations). We explored very different kinds of prompts for both GPT-2 and GPT-3. A complete account of all the prompts we tried can be found in Table 9 in the supplementary material. Table 2 here shows the prompts that gave the best results, consequently being used in the experiments.

heading: 4 Results

text: We first report the results of the main experimental setting that compares humans, language models and other systems, restricted to 8 concepts (test examples with the empty string as output not included in the means). More extensive results with a larger number of concepts at the end of section.

heading: 4.1 Comparison against Humans using 8 Concepts

text:
We take the results of each of the learners in the previous section. For humans, we aggregate the results into a single value per concept, batch and test example, by calculating the average of the 30 human subjects. Performance is represented using learning curves and characteristic curves. A learning curve depicts the variation of accuracy according to the number of examples, which in our case is grouped into the batches WS, AS I and AS II. Characteristic curves depict the variation of accuracy in terms of the complexity level of the concepts: VL-C, L-C, H-C and VH-C. Accuracies are always reported as a percentage of the test examples that are correctly predicted.
We start with the analysis for the eight GPT-3 parameter configurations. The learning curves in Figure 1a are clearly increasing in general. The first observation is that the systems with the temperature set to 0 obtain better performance than those with the temperature set to 1. There is no clear evidence that larger models in terms of the number of parameters present a better performance for this kind of patterns, which are solely based on simplicity and not other priors. Regarding the characteristic curves in Figure 1b, the GPT-3 systems show a decreasing trend in performance as the complexity increases. Once again, there is no clear pattern in terms of the number of the parameters. Nevertheless, GPT-3D-T0 is the best method, and both GPT-3D versions (the largest models) show their advantage in the very few shot case (WS) and low complexity (VL-C). In other words, more parameters seem to make a difference for easier concepts with fewer examples.
Figure 2a shows the learning curves for humans, MH, the two variations for Louise and a selection of GPT methods. GPT-3D-T0 turns out to be the system with the highest average performance for every teaching batch. Louise’s variants follow, with a similar performance between them. Humans and GPT-2-ens go next, with similar general performance in the experiment. GPT-2-exp remains slightly under the average performance of the ensemble (majority) approach. Finally, MH is the poorest system, especially for WS, the case with very few examples. For all systems, the accuracy increases as additional examples are provided (AS I and AS II).
Figure 2b shows the characteristic curves. We see curves cross more often, but overall the order of the learners in terms of performance is similar to the learning curves. The evolution of performance in terms of complexity is generally decreasing, with the best results clearly being for concepts of
very low complexity (VL-C). Looking into the results in more detail (see supplementary material), we find different reasons for the small increase in the last group (VH-C), but it is mostly explained because the chosen test examples have short outputs for these two concepts. This increase for VH-C disappears when experiments are performed with a large number of concepts, as we discuss below.
Explanations for the given predictions may be a good way of better understanding whether alignment has taken place, beyond accuracy. When looking for explanations, we tried many kinds of prompts for the GPT systems, as discussed in the supplementary material. However, we did not manage to get good explanations. For Louise, despite being interpretable, the solutions usually had too many clauses to serve as explanations. Only for MH the solution programs are easily interpretable by humans, assuming some knowledge of functional programming, but MH was seldom correct. Because of all this, we will pay more attention to the explanations given by humans in the questionnaire.
We consider a correct explanation if, after inspection by us, the whole behaviour of the concept is well explained and is totally correct. We consider it incomplete if it is a partial account of the concept. We consider it incorrect if any part of it does not correspond to the concept. Note that the explanation might be wrong and the inferences correct, and the other way round. As observed in Figure 3a, the percentage of correct natural language explanations decreases as the complexity of the concepts increases. On the other hand, Figure 3b shows how the average size of the correct explanations increases with the complexity of the concepts (size is measured as the number of characters in the natural language explanation given by humans). Remarkably, the average size of the different types of explanations present the opposite behaviour to performance considering the concept complexity bins in Figure 2b. Even for incorrect explanations, the perceived complexity in terms of the size of the explanation inversely corresponds to the accuracy results.
In the previous plots, we have compared learners by batches of examples or complexity of the concepts. However, if we want to have a more fine-grained metric of how much similar the systems are, we can analyse if they succeed on the same examples. This is what we see in Figure 4, showing the error disagreement (proportion of cases where one method is right and the other is wrong, or vice versa) (Figure 4a) and the Phi coefficient (Figure 4b) taking all the examples individually. The matrix for error disagreement strongly depends on performance, so the results for Phi coefficient (Pearson correlation) are more informative. We see that approaches of the same family usually have high correlations. When compared with humans, GPT-3D-T0 and Louise-exp are the most similar ones.

heading: 4.2 Results for Language Models for Larger Concept Samples

text:
The supplementary material includes more details of the comparison between humans, language models and the program induction systems. While comparison against humans is limited by the number of concepts that humans could attempt in a single form, we do not have these limitations when using language models. In particular, the supplementary material includes concepts with and without loops. We see that GPT-3 is better for concepts without loops, especially when sufficient examples are given in phase three (WS + AS I + AS II).
Here we just report the results for much larger samples. Table 3 includes the results considering 300 concepts (shown also in the supplementary material with other parameter sizes in Tables 15, 16, 17 and 18). What we see is a much more gradual increase of performance as more information is given through the three batches, and also a decrease of performance as the concepts are more complex.
The supplementary material also includes results for the majority and random baselines (discarding spurious effects) and experiments not using machine teaching. All the code and experimental data can be found at https://github.com/gonzalojaimovitch/think-big-teach-small.

heading: 5 Discussion

text:
We can now answer the questions we formulated in Section 3. We see that only for easy concepts (VL-C), the simplicity prior leads to example sets that humans, Louise and some GPT models can identify consistently (Q1, Q3). For more complex concepts the behaviour is more erratic, and may depend on spurious factors, as the explanations are incorrect for those very hard cases having some accuracy. The learning curves are increasing for all systems, as more examples are given (Q2). The minimal set is rarely sufficient, which means that no system prior is perfectly aligned with the simplicity prior that derives from P3 program size. This is expected as the internal representations of these systems are different. Both humans and GPT have priors to capture both extrinsic and intrinsic patterns, as language models have been trained using natural language texts created by humans. This seems consistent with large language models having a comparable performance to humans. However, this would not explain cases in which GPT-3 outperforms humans, unless (1) our population behaves differently (worse) than the population behind the training corpora used for learning them, and/or (2) GPT-3 has generalised some very abstract latent concepts that are applicable for simple intrinsic patterns, where GPT-3 seems to excel (VL-C). Finally, MH is the system that behaves most like an ideal learner in our setting, but with very limited capability. The difference in the performance between MH and Louise might be due to the differences in their background knowledge.
Considering both matrices in Figure 4, GPT-3D-T0 shows a similar pattern of behaviour (correct/incorrect results) with respect to humans in comparison with the rest of the analysed systems (Q4). This means the extra performance is not explained by being more deviated from human priors, but perhaps because of higher robustness. When we look at architecture size, we do not see a monotonic relation between results and number of parameters (Q5), but GPT-3D, for the two variants that we have studied, achieves the best results for simple concepts. This may suggest that larger models can be better at capturing patterns that depend on simplicity priors, distilling some kind of Occam’s razor for few-shot inference. However, some other experiments in the supplementary material, especially those concepts including loops, suggest that larger models in the future may not be able to solve complex algorithmic concepts, with this distillation being limited to sequential (if-then-else) algorithmic patterns. New paradigms beyond transformers may be needed in the future.
Explanations should give us more insight about what these systems are doing. Unfortunately, we were not able to find a working prompt so that GPT-3 could generate good explanations for the learnt concepts (Q6). Regarding the explanations provided by humans, the results suggest that the higher the complexity, the higher the size of the correct explanations to cover the learnt concepts. Furthermore, as complexity increases, the percentage of correct explanations decreases.
There are some limitations in this work. First, while we use MH as a system that would only be dominated by the simplicity priors, it is not very capable, and it is only useful to disentangle the behaviour for very easy concepts. Louise adds robustness to this analysis but it has more dependency on background knowledge. More inductive programming systems could help in this analysis. Second, the use of humans for comparison limits the number of concepts and test examples that we can do for a reasonable questionnaire. This has been partially mitigated by more concepts being explored for GPT-3 (section 4.2 and supplementary material), but the analysis of different samples according to other characteristics (not only loops but the frequencies of symbols or the lengths of the strings) could further characterise the performance. Third, the machine teaching setting we have used assumes perfect alignment and perfect identification, and this does not help in the understanding of how to provide further examples (AS I and AS II). Similarly, for compatibility with the original teaching book, in case of equal l, ties were broken lexicographically, which adds more dependence on P3. Fortunately, the machine teaching setting can be extended probabilistically for the teaching dimension (Hernández-Orallo and Telle, 2020) (with more evidence increasing the confidence of a correct identification), and a similar extension for the teaching size could be used for this analysis.
The main take-away of this paper is a new approach, based on machine teaching, to analyse the power of language models under few-shot inference. Because of the complexity of transformer architectures, the diversity of the training data used for language models, and their use as pre-trained models through very different kinds of prompts, we require empirical yet well-founded ways to analyse how much evidence is needed for these systems to make the right extrapolations. This analysis boils down to determining the priors that pretrained models have distilled and the bias they represent. Overall, machine teaching can bring very important insights about what language models can do, what their acquired priors are, and the number —or the size— of examples they need.

heading: Acknowledgments

text:
We thank OpenAI for giving us early access to GPT-3 through their API. We thank support by the EU (FEDER) and Spanish grant RTI2018-094403-B-C32 funded by MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”, Generalitat Valenciana under grant PROMETEO/2019/098 and INNEST/2021/317 (Project cofunded by the European Union with the "Programa Operativo del Fondo Europeo de Desarrollo Regional (FEDER) de la Comunitat Valenciana 2014-2020”), EU’s Horizon 2020 research and innovation programme under grant agreement No. 952215 (TAILOR), the FLI under grant RFP2-152, and US DARPA HR00112120007 (RECoG-AI). We thank the anonymous reviewers for their comments and interaction during the discussion process. We are grateful to Dan Hendricks, Fernando Martínez-Plumed and Richard Evans for useful discussions, Carlos Monserrat for some assistance with the plots, Lidia Contreras-Ochando for her help with MagicHaskeller and María José Ramírez-Quintana for their help with questions about visualisation, inductive programming and Louise. We also thank Stephen Muggleton and Stassa Patsantzis for their help with Louise. Finally, we thank the participants who completed the questionnaire.
Funding Transparency Statement
All sources of funding have been included in the acknowledgments. None of the authors have financial relationships with entities that could potentially be perceived to influence the objectivity of the paper.

title: Think Big, Teach Small: Do Language Models Distil Occam’s Razor?


summaries: This study aims at distilling feature representations (mainly the last convolution layer) into robust and non-robust features using information bottleneck. For every layer, the feature maps are partitioned into robust and non-robust for each individual example. After finding this, the paper first illustrates that selectively propagating robust features of adversarial examples built for a robust network will not degrade the accuracy but propagating non-robust features does which to some extent illustrates that the information bottleneck method has correctly identified the robust and non robust features. They also illustrate that the adversarial class label has high correlation to the Non-robust features.

summaries: This paper categorizes the intermediate features of a DNN into robust ones and non-robust ones. Through visualization, the non-robust ones display different semantic information from the true label, and thus potentially leads to the existence of AE.  The paper proposes a new AE generation algorithm targeting the non-robust feature, and beats the common attack baselines.

summaries: This work utilizes an Information Bottleneck (IB) to distill robust and non-robust features in neural networks (NNs). Experiments are performed to show that the distilled features are correlated with adversarial prediction and are human-perceptible. Finally, an attack mechanism intensifying the gradient of non-robust features is proposed.

summaries: The paper proposes to explicitly separate intermediate representation into robust and non-robust categories by information bottleneck, and demonstrate that the non-robust feature is related to adversarial prediction. An attack mechanism based on enforcing non-robust features gradient is proposed. The experimental results show the effectiveness of the proposed approach.

summaries: This paper focuses on robust and non-robust features in adversarial examples. This paper proposes a method of distilling robust and non-robust features via information bottleneck. Through analysis, this paper has shown the high correlation between the distilled features and the adversarial prediction. Based on that, the authors design a new attack that intensifies the gradient of non-robust features.

abstractText: Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields. Recent works have argued that the existence of the robust and non-robust features is a primary cause of the adversarial examples, and investigated their internal interactions in the feature space. In this paper, we propose a way of explicitly distilling feature representation into the robust and non-robust features, using Information Bottleneck. Specifically, we inject noise variation to each feature unit and evaluate the information flow in the feature representation to dichotomize feature units either robust or non-robust, based on the noise variation magnitude. Through comprehensive experiments, we demonstrate that the distilled features are highly correlated with adversarial prediction, and they have human-perceptible semantic information by themselves. Furthermore, we present an attack mechanism intensifying the gradient of non-robust features that is directly related to the model prediction, and validate its effectiveness of breaking model robustness.

authors:
C. Szegedy
W. Zaremba
I. Sutskever
J. Bruna
D. Erhan
I. Goodfellow
R. Fergus

title: Intriguing properties of neural networks

venue: International Conference on Learning Representations (ICLR), 2014.

authors:
I.J. Goodfellow
J. Shlens
C. Szegedy

title: Explaining and harnessing adversarial examples

venue: International Conference on Learning Representations (ICLR), 2015.

authors:
P.-Y. Chen
H. Zhang
Y. Sharma
J. Yi
C.-J. Hsieh

title: Zoo: Zeroth order optimization based blackbox attacks to deep neural networks without training substitute models

venue: ACM Workshop on Artificial Intelligence and Security, 2017.

authors:
N. Carlini
D. Wagner

title: Towards evaluating the robustness of neural networks

venue: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57, IEEE Computer Society, 2017.

authors:
N. Papernot
P. McDaniel
I.J. Goodfellow
S. Jha
Z.Y. Celik
A. Swami

title: Practical black-box attacks against machine learning

venue: ACM ASIA Conference on Computer and Communications Security (ACM ASIACCS), 2017.

authors:
K. Eykholt
I. Evtimov
E. Fernandes
B. Li
A. Rahmati
C. Xiao
A. Prakash
T. Kohno
D. Song

title: Robust physical-world attacks on deep learning visual classification

venue: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1625–1634, 2018.

authors:
A. Madry
A. Makelov
L. Schmidt
D. Tsipras
A. Vladu

title: Towards deep learning models resistant to adversarial attacks

venue: International Conference on Learning Representations (ICLR), 2018. 10

authors:
G. Apruzzese
M. Colajanni
L. Ferretti
M. Marchetti

title: Addressing adversarial attacks against security systems based on machine learning

venue: International Conference on Cyber Conflict (CyCon), vol. 900, pp. 1–18, IEEE, 2019.

authors:
X. Wang
J. Li
X. Kuang
Y. an Tan
J. Li

title: The security of machine learning in an adversarial setting: A survey

venue: Journal of Parallel and Distributed Computing, vol. 130, pp. 12 – 23, 2019.

authors:
Y.E. Sagduyu
Y. Shi
T. Erpek

title: Iot network security from the perspective of adversarial deep learning

venue: International Conference on Sensing, Communication, and Networking (SECON), pp. 1–9, 2019.

authors:
I. Rosenberg
A. Shabtai
Y. Elovici
L. Rokach

title: Adversarial learning in the cyber security domain

venue: arXiv preprint arXiv:2007.02407, 2020.

authors:
A. Shafahi
W.R. Huang
C. Studer
S. Feizi
T. Goldstein

title: Are adversarial examples inevitable

venue: International Conference on Learning Representations (ICLR), 2018.

authors:
L. Schmidt
S. Santurkar
D. Tsipras
K. Talwar
A. Madry

title: Adversarially robust generalization requires more data

venue: Neural Information Processing Systems (NeurIPS), 2018.

authors:
D. Tsipras
S. Santurkar
L. Engstrom
A. Turner
A. Madry

title: Robustness may be at odds with accuracy

venue: International Conference on Learning Representations (ICLR), 2019.

authors:
A. Ilyas
S. Santurkar
D. Tsipras
L. Engstrom
B. Tran
A. Madry

title: Adversarial examples are not bugs, they are features

venue: Neural Information Processing Systems (NeurIPS), vol. 32, 2019.

authors:
L. Engstrom
J. Gilmer
G. Goh
D. Hendrycks
A. Ilyas
A. Madry
R. Nakano
P. Nakkiran
S. Santurkar
B. Tran
D. Tsipras
E. Wallace

title: A discussion of ’adversarial examples are not bugs, they are features

authors:
N. Tishby
F.C. Pereira
W. Bialek

title: The information bottleneck method

venue: Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pp. 368–377, 1999.

authors:
A. Alemi
I. Fischer
J. Dillon
K. Murphy

title: Deep variational information bottleneck

venue: International Conference on Learning Representations (ICLR), 2017.

authors:
K. Schulz
L. Sixt
F. Tombari
T. Landgraf

title: Restricting the flow: Information bottlenecks for attribution

authors:
M.D. Zeiler
R. Fergus

title: Visualizing and understanding convolutional networks

venue: European Conference on Computer Vision (ECCV), pp. 818–833, Springer, 2014.

authors:
J. Yosinski
J. Clune
T. Fuchs
H. Lipson

title: Understanding neural networks through deep visualization

venue: International Conference on Machine Learning Workshop on Deep Learning, Citeseer.

authors:
D. Bau
B. Zhou
A. Khosla
A. Oliva
A. Torralba

title: Network dissection: Quantifying interpretability of deep visual representations

venue: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6541–6549, 2017.

authors:
C. Olah
A. Satyanarayan
I. Johnson
S. Carter
L. Schubert
K. Ye
A. Mordvintsev

title: The building blocks of interpretability

authors:
L. Engstrom
A. Ilyas
S. Santurkar
D. Tsipras
B. Tran
A. Madry

title: Adversarial robustness as a prior for learned representations

venue: arXiv preprint arXiv:1906.00945, 2019.

authors:
C. Olah
A. Mordvintsev
L. Schubert

title: Feature visualization

authors:
S. Kullback
R.A. Leibler

title: On information and sufficiency

venue: The annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951.

authors:
C.-J. Simon-Gabriel
Y. Ollivier
L. Bottou
B. Schölkopf
D. Lopez-Paz

title: First-order adversarial vulnerability of neural networks and input dimension

venue: International Conference on Machine Learning (ICML), pp. 5809–5817, PMLR, 2019.

authors:
C. Qin
J. Martens
S. Gowal
D. Krishnan
K. Dvijotham
A. Fawzi
S. De
R. Stanforth
P. Kohli

title: Adversarial robustness through local linearization

authors:
K. Simonyan
A. Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: International Conference on Learning Representations (ICLR), 2015. 11

authors:
S. Zagoruyko
N. Komodakis

title: Wide residual networks

venue: British Machine Vision Conference (BMVC), 2016.

authors:
L. Van der Maaten
G. Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research, vol. 9, no. 11, 2008.

authors:
A. Krizhevsky
G. Hinton

title: Learning multiple layers of features from tiny images

authors:
Y. Netzer
T. Wang
A. Coates
A. Bissacco
B. Wu
A.Y. Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
Y. Le
X. Yang

title: Tiny imagenet visual recognition challenge

authors:
R. Fong
A. Vedaldi

title: Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks

venue: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8730–8738, 2018.

authors:
S. Kaur
J. Cohen
Z.C. Lipton

title: Are perceptually-aligned gradients a general property of robust classifiers

venue: NeurIPS Workshop: Science Meets Engineering of Deep Learning (NeurIPS SEDL), 2019.

authors:
D. Erhan
Y. Bengio
A. Courville
P. Vincent

title: Visualizing higher-layer features of a deep network

venue: University of Montreal, vol. 1341, no. 3, p. 1, 2009.

authors:
A. Nguyen
J. Yosinski
J. Clune

title: Understanding neural networks via feature visualization: A survey

venue: Explainable AI: interpreting, explaining and visualizing deep learning, pp. 55–76, Springer, 2019.

authors:
A. Mahendran
A. Vedaldi

title: Understanding deep image representations by inverting them

venue: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5188–5196, 2015.

authors:
A. Kurakin
I.J. Goodfellow
S. Bengio

title: Adversarial machine learning at scale

authors:
F. Croce
M. Hein

title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameterfree attacks

venue: International Conference on Machine Learning (ICML), 2020.

title: Minimally distorted adversarial examples with a fast adaptive boundary attack

venue: Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 2196–2205, PMLR, 13–18 Jul 2020.

authors:
H. Zhang
Y. Yu
J. Jiao
E. Xing
L. El Ghaoui
M. Jordan

title: Theoretically principled trade-off between robustness and accuracy

venue: International Conference on Machine Learning (ICML), pp. 7472–7482, PMLR, 2019.

authors:
Y. Wang
D. Zou
J. Yi
J. Bailey
X. Ma
Q. Gu

title: Improving adversarial robustness requires revisiting misclassified examples

venue: International Conference on Learning Representations (ICLR), 2020.

authors:
N. Inkawhich
K. Liang
L. Carin
Y. Chen

title: Transferable perturbations of deep feature distributions

authors:
J.-H. Jacobsen
J. Behrmann
R. Zemel
M. Bethge

title: Excessive invariance causes adversarial vulnerability

authors:
V.A. Banks
K.L. Plant
N.A. Stanton

title: Driver error or designer error: Using the perceptual cycle model to explore the circumstances surrounding the fatal tesla crash on 7th may 2016

venue: Safety Science, vol. 108, pp. 278–285, 2018.

authors:
V. Antun
F. Renna
C. Poon
B. Adcock
A.C. Hansen

title: On instabilities of deep learning in image reconstruction and the potential costs of ai

venue: Proceedings of the National Academy of Sciences, 2020. 12

heading: 1 Introduction

text:
Deep neural networks (DNNs) have achieved remarkable performances in a wide variety of machine learning tasks. Despite the breakthrough outcomes, DNNs are easily fooled from adversarial attacks, with crafted perturbations [1, 2, 3, 4, 5, 6, 7]. These perturbations are imperceptible to human eyes, but simply adding them to clean images (i.e., adversarial examples) can effectively deceive classifiers. Such a vulnerability affects security problems [8, 9, 10, 11], bringing in the weak reliability of DNNs.
Previous works have broadly investigated the reason for the widespread of such adversarial examples. Goodfellow et al. [2] have argued that adversarial vulnerability is induced from the excessive linearity nature of DNNs in high-dimensional spaces. Several works [1, 12] have regarded the primary cause of the adversarial examples as statistical variation with aberrations in data manifold. Schmidt et al. [13] have suggested that the pervasiveness of the examples should not be considered as a drawback of training methods for DNNs, since the available dataset may not be large enough to train them robustly.
In recent years, Tsipras et al. [14] have suggested an intriguing analysis that the disagreement between standard and adversarial accuracy stems from differently trained feature representation. In this literature, Ilyas et al. [15] further have demonstrated the adversarial examples are inevitable results of standard supervised training and arisen from well-generalized features in the dataset. They have analyzed the adversarial examples are originated from brittle and unintelligible features (i.e., non-robust features) that are arbitrarily manipulated with the imperceptible noise, and shown that the
∗Equal contribution. † Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
robust features still can provide precise accuracy even in the existence of adversarial perturbation. They have argued that the non-robust features cannot show reliable accuracy in the adversarial setting and could provoke incomprehensible properties.
Nonetheless, the underlying reason for the existence and pervasiveness of adversarial examples cannot derive common consensus in the research field and still remains unclear [16]. To clarify where the adversarial brittleness truly comes from, we need to figure out how the robust and non-robust features in data manifold subtly manipulate feature representation and fool model prediction, by directly handling them in the feature space. To address it, we propose a way to precisely distill intermediate features into robust and non-robust features by employing Information Bottleneck (IB) [17, 18, 19]. In the sense that semantic information is included in the units of intermediate feature representation [20, 21, 22, 23], we utilize the bottleneck to regulate the information flow in the feature space by explicitly adding noise to the feature units. Then, we estimate how each feature unit contaminated with the noise affects model prediction with assigned information. Based on the prediction sensitivity of the noise intervention, we assort the feature units either robust or brittle, and disentangle the feature representation into robust or non-robust features, respectively.
Through extensive analysis of the distilled features, we corroborate that the pervasiveness of the adversarial brittleness is derived from the non-robust features, and they have a high correlation with the adversarial prediction. In addition, in order to understand the semantic information of distilled features, we directly visualize them in the feature space and provide their visual interpretation. Consequently, we reveal that both of the robust and non-robust features indeed have semantic information in terms of human-perception by themselves. Based on our observation, we theoretically describe the negative impact of the non-robust features for the model prediction and introduce an approach of amplifying the gradients of non-robust features to break the model prediction.
In this paper, our contributions can be summarized into three-fold as follows:
• We propose a novel way to explicitly distill intermediate features into the robust and nonrobust features using Information Bottleneck, and interpret the disentangled features in terms of human-perception by directly visualizing them in the feature space.
• By analyzing how the distilled features affect the intermediate feature representation under adversarial perturbation, we demonstrate that the non-robust features are highly correlated with the adversarial prediction.
• We present an attack mechanism manipulating the non-robust features by strengthening their gradients, and validate its effectiveness of breaking model prediction.

heading: 2 Distilling Robust and Non-robust Features in Intermediate Feature Space

text:
Problem Setup and Notations. Let X denote clean images and Y denote (one-hot encoded) target labels corresponding to the clean images. Then, adversarial examples Xadv can be created by the following equations: max
δ E(X,Y )[L(f(X + δ), Y )], where δ denotes an adversarial perturbation,
and L denotes a pre-defined loss for machine learning tasks. The adversarial examples can be made by Xadv = X + δ. When a given model f is adversarially trained against PGD attack [7], it can be written as follows:
min w max ‖δ‖∞≤γ
E(X,Y ) [L (f(X + δ), Y )] , (1)
where w represents the parameters of f , which are learned to be robust against adversarial attacks. Here, ‖·‖∞ ≤ γ describes L∞ norm, and γ-ball means the perturbation magnitude. In this paper, we adversarially train the model f on γ = 0.03 for the standard adversarial attack. Note that once adversarially trained, the parameters of the model f are no longer covered.1
As notation of variables that we will use in this paper, Z and Z̄ indicate the intermediate features of the model f such that Z = fl(X) and Z̄ = fl(Xadv), where fl(·) describes l-th layer outputs of the given model. Similarly, fl+(·) represents subsequent network after the l-th layer, thus intermediate features can be propagated to the last output layer, such that Ŷ = fl+(Z) and Ŷadv = fl+(Z̄). The
1Previous works [14, 15] have demonstrated that the distinction between the robust and non-robust features arises in adversarial settings. In the sense that adversarially trained networks learn robust representation [24], we set the robust classifier as default. Please see the analysis of the standard training in Appendix F.
model f can be expressed as f = fl+ ◦ fl, satisfying Ŷ = f(X) for the given clean images X . Also, Ŷadv = f(Xadv) is denoted by model propagation of the adversarial examples Xadv. Note that we designate l-th layer as the last convolutional layer, and regard l+ as the rest of the layers in the model.

heading: 2.1 Information Bottleneck for Distilling Informative Features

text:
In adversarial settings, we focus on separating robust and non-robust features in the intermediate layer. Recall that robust features are literally robust on the noise (variation) and invariant to the existence of the adversarial perturbation, but non-robust features are not. Our approach aims to distill feature units that affect target prediction under the noise perturbation in the intermediate feature space.
We follow that the semantic information is inherently included in the feature units of DNNs [20, 22, 25]. From this perspective, we utilize Information Bottleneck (IB) to distill the robust and non-robust features on the given intermediate features Z. Information Bottleneck [17, 18] proposed to encode maximally informative representation for target labels, restraining input information, concurrently. Using the bottleneck, we suggest a way to assess feature importance and quantify information flow for the target prediction. The objective function of IB can be written as follows:
max Z
I(Z, Y )− βI(Z,X), (2)
where I denotes mutual information, and β represents the degree of restraining input information. The first term I(Z, Y ) allows the intermediate features Z to be predictive on the target label Y , and the second term I(Z,X) encourages Z to compress the information of the given images X in the bottleneck. Here, the second term requires a true feature probability p(Z) = ∫ X p(Z | X)p(X)dX to expand it, but it is computationally intractable due to a high dimensional dependency of the dataset probability p(X). Thus, several works [18, 19] modified the IB’s objective function to make it possible to learn DNNs without the true feature probability as follows: minLCE + βLI (see Appendix B). In this formulation, LCE indicates cross-entropy loss, and LI represents information loss computed by KL divergence [26] between a feature likelihood p(Z | X) and an approximate feature probability q(Z). It is radically a closed-form approximation for the true feature probability p(Z).
Firstly, we deliberately inject noise variation into Z to estimate the prediction sensitivity of each feature unit along the noise intervention. To do so, we newly design an approximate feature probability using noise variation σ, such that qσ(Z) = N (fl(X), σ2). Then, we sample random variables from qσ(Z) and define informative features ZI as follows:
ZI = fl(X) + σ · , (3) where ZI ∼ qσ(Z). Note that the operator · denotes Hadamard product, and stands for Gaussian noise sampled from N (0, I). Here, the noise variation measures a correlation between intermediate features and model prediction based on the fact that robustness means high correlation on model prediction and non-robustness are opposite [14, 15] in adversarial settings. Since the correlation E(X,Y )[Y ·f(X)] in output layer can be expressed as a variance measure Cov(Y, f(X)), we consider the correlation in intermediate layer as the noise variation. From this perspective, if a feature unit is
highly predictive despite large noise variation (high correlation), the unit can robustly predict target labels, while a non-robust unit cannot.
Once the informative features ZI are acquired from the noise variation σ, we propagate ZI to the last output layer and estimate the feature importance of each unit for model prediction. Then, we deal with the information loss LI in order to alleviate feature heterogeneity between Z and ZI . Through aforementioned modification [18, 19], our objective function can be written as follows:
min σ LX(σ) = −Y log fl+(fl(X) + σ · )︸ ︷︷ ︸
LCE
+β DKL[p(Z | X) || qσ(Z)]︸ ︷︷ ︸ LI , (4)
where the feature likelihood p(Z | X) is set to N (fl(X), σ2z). Here, σz indicates inherent feature variation of the intermediate features Z for each unit. The second term LI makes the informative features ZI resemble the intermediate features Z, while minimizing the cross-entropy LCE . This second term can be written as LI = 12 ∑C k=1[ σ2zk σ2k + log σ2k σ2zk − 1], where k denotes an index of the noise variation σ = [σ1, σ2, · · · , σC ] (the optimizing parameters) and the feature variation σz = [σz1 , σz2 , · · · , σzC ] (the given parameters). Here, only of the variation σ is updated such that σ ← σ − ∂∂σLX(σ). In brief, we summarize overall procedure of our bottleneck concept in Fig. 1. Moreover, we mention that β in Eq. (4) controls the amount of information that flows into the feature representation. Specifically, when β is set to zero, IB loss is equivalent to cross-entropy loss, which means that ZI can accommodate even unimportant features to predict target labels. In contrast, excessively large β only focuses on compressing input information, thus IB may cannot filter out important features to predict target labels. Accordingly, we empirically control β to distill informative features ZI based on the noise variation σ (Please see section 3.4 for analysis of information flow).

heading: 2.2 Separating Informative Features by Tolerance of Feature Variation

text:
After optimizing the informative features ZI , we compare the noise variation σ for the informative feature units and dichotomize each unit either robust or non-robust based on their prediction sensitivity. We set the criterion for comparison as T = max(σ2z). Here, T represents the maximum tolerance of the noise variation. It is a reasonable choice to set T as a criterion, because it indicates the maximal variation with respect to the changes of the given image X , in the feature space. In the following procedure, we explicitly disentangle intermediate features Z into the robust Zr and non-robust features Znr.
Firstly, once the noise variation σ is larger than the maximum tolerance T in specific units, it indicates that their corresponding features are highly predictive on the model prediction, despite the noise intervention. Thus, we define their conjunction as robust features. On the other hand, if the variation of a specific unit is smaller than T , their corresponding features can be represented as non-robust features. This is because the small variation behaves as a strict restriction to retain model prediction of target labels. We assume that once a strong adversarial perturbation comes in, the feature variation of non-robust features becomes to be larger than acceptable tolerance, thereby easily breaking the model classifier and leading to misclassified prediction. In this respect, we sort the noise variation according to their magnitude, and cluster them by assigning robust or non-robust channel indexes to each feature unit. The robust channel index, ir = [ir1 , ir2 , · · · , irC ] can be computed as follows:
irk = 1(σ 2 k > T ) = { 1 σ2 > T 0 σ2 ≤ T , (5)
where 1(·) represents the indicator function. The non-robust channel index inr is simply reversed from the robust channel index such that inrk = 1 − irk . Then, we estimate robust features Zr by multiplying the robust channel index to the intermediate features element-wisely such that Zr = ir ·Z. Similarly, non-robust features Znr are presented as Znr = inr · Z. In this way, the intermediate features Z are fully disentangled into the two types of feature representation satisfying Z = Zr+Znr.
To sum it up, we regard the robust features Zr that have the larger noise variation as invariant features from the adversarial perturbation. On the other hand, non-robust features Znr are considered as easily manipulated features, which harmonize the smaller noise variation. Now, we analyze their impacts to the robustness by expanding the model prediction of ZI to Taylor approximation (see Appendix C.)
with its convergence of local minima [27, 28] as follows:
fl+(fl(X) + σ · ) = fl+(fl(X) + σr · ) + [ ∂
∂σr fl+(fl(X) + σr · ) ]T σnr︸ ︷︷ ︸
∆
, (6)
where robust noise variation σr = ir · σ and non-robust noise variation σnr = inr · σ. Since the variation of the robust features does not degrade the model prediction, when σnr is small enough, the erroneous term ∆ in Eq. (6) closes to zero. That is, the model retains having significant robustness against the adversarial perturbation interrupting robust channel index in Z. Conversely, once we force σnr to increase, its output becomes inaccurate for the robust prediction (i.e., first term in Eq. (6)) due to high ∆. Here, we theoretically demonstrate how the brittleness of non-robust features affects the model robustness. We will thoroughly analyze the properties of the two distilled features by empirically showing the robustness of Zr and brittleness of Znr in the following sections.

heading: 3 Analysis of Distilled Features and Visual Interpretation

heading: 3.1 Property of Distilled Feature Units under Adversarial Perturbation

text:
After we distill the robust and non-robust features using the bottleneck concept, our next question is How will the target prediction change under the adversarial attacks? In our posit, if the bottleneck successfully disentangles the robust and non-robust channel index (i.e., ir and inr) from the given examples, we should identify the consequential classification accuracy changes under the adversarial perturbation. That is, after applying the robust index to the attacked feature representation, the selected adversarial features with ir denoted by Z̄r (i.e., Z̄r = ir · Z̄) should have invariant accuracy changes for the target labels. Here, ir is robust channel index obtained from the given clean examples X , and Z̄ represents the intermediate features of the adversarial examples, which means Z̄ = fl(Xadv). Contrarily, the selected features satisfying Z̄nr = inr · Z̄ will show inaccurate accuracy due to their brittleness under the existence of adversarial perturbation.
In Table 1, we analyze evaluation results of the disentangled features under standard attack algorithms [2, 4, 7] in publicly available datasets [32, 33, 34]. As aforementioned, we apply the robust and non-robust channel index optimized from the clean examples to the adversarial features Z̄, and estimate their accuracy (i.e., fl+(Z̄r) and fl+(Z̄nr)). As in the table, Z̄r still shows constant robust accuracy regardless of the adversarial perturbation, even in the high-confidence adversarial attack [4]. On the other hand, we can find that the classification accuracy of Z̄nr steeply degrades as the attacks get stronger, which coincides with the properties of the robust and non-robust features. To further support our experiments, we illustrate the correlation between the disentangled features and true labels, using 2D t-SNE plot [31]. In the case of clean examples as in Fig. 2(a), the robust features Zr exhibit separable clusters on the target labels, while the non-robust features Znr show a partially disorganized tendency. When adversarial perturbation [7] exists, the attacked features Z̄nr represents more collapsed t-SNE visualization as shown in Fig. 2(b). Notably, we can observe that Z̄r still sustain highly clustered results even in the attacked condition.

heading: 3.2 Feature Visualization of Robust and Non-robust Features

text:
We have identified the existence of the robust and non-robust features using the bottleneck. Then, we wonder about a way of interpreting the semantic information in the feature space with respect to human-perception. Analyzing semantic representation of DNNs is a wide research area to understand their decision [20, 22, 35]. In adversarial settings, several studies [14, 36] argued that a robust classifier has more meaningful (i.e., perceptually-aligned) loss gradients in the input space. Engstrom et al. [24] further endeavored to interpret robust feature representation using feature visualization [25, 37, 38]. In this manner, we explore whether the disentangled feature representation from the bottleneck indeed has human-perceptible information in the intermediate feature space.
Feature visualization is an optimization-based method that maximizes specific activation of feature units [38, 39], such that X ′ = argmaxX(a l i(X)), where a l i(·) indicates feature activation of i-th unit in the l-th layer. To understand what conjunction of the robust and non-robust feature units truly interacts with the target labels, we optimize each distilled feature and create their visual explanations. We adopt direct visualization method [25] that has various regularization techniques (e.g., frequency penalization and transformations) to yield better representative visual quality.
We employ iar and i a nr for the adversarial examples Xadv , which is obtained by optimizing LXadv (σ) instead of Eq. (4). Then, we define Z̄ar and Z̄ a nr as robust and non-robust features of the adversarial examples, satisfying Z̄ar = i a r · Z̄ and Z̄anr = ianr · Z̄. After distilling robust and non-robust features with their corresponding index, we maximize the selected feature unit activation, respectively. The feature visualization results of distilled features are illustrated in Fig. 3.
As in the figure, we can clearly recognize the semantic information of true labels Y and attacked predictions Ŷadv in the intermediate features (Z and Z̄). Interestingly, what we can observe is: (i) the distilled features have semantic information by themselves and maintain their information, even under the adversarial perturbation, (ii) when the adversarial perturbation exists, the brittleness of non-robust features is intensified and reflected onto Z̄. Thus, the visualization of Z̄ and Z̄anr looks similar, and they manipulate the target prediction to same adversarial prediction. Unlike the previous work [15] that has argued the non-robust features solely have incomprehensible property, the visualization of the distilled features from our bottleneck represent recognizable outputs even for the adversarial examples, and provides a decisive key to interpret the cause of adversarial examples.

heading: 3.3 Adversarial Prediction is Highly Correlated with Non-robust Features

text:
We have observed that the non-robust features optimized from the bottleneck are brittle and easily manipulated under the adversarial perturbation, while robust features maintain substantial prediction results for the target labels. Then, if the primary cause of the adversarial examples indeed belongs to non-robust features, it is natural to examine the correlation between the classification outputs of the non-robust features and the adversarial prediction induced by adversarial attacks. Accordingly, we identically apply our IB loss on the adversarial examples Xadv [2, 7, 4], and find their corresponding robust and non-robust channel index using Eq. (5).
To enlighten the correlation of non-robust features and the adversarial prediction, we evaluate the model prediction of Z̄anr for the attacked labels Ŷadv that can be written as follows: Ŷ a nr = fl+(Z̄ a nr). We set the condition of Ŷadv as successfully attacked labels (i.e., Y 6= f(Xadv)) to definitely show the relationship between the prediction Ŷ anr of non-robust features and the adversarial prediction Ŷadv of adversarial examples. In Table 2, we summarize the accuracy of Ŷ anr for the successfully attacked label Ŷadv in standard attack methods. Generally, we can observe that the non-robust features are highly predictive on Ŷadv in standard low dimensional datasets such as CIFAR-10 and SVHN. Even in a large dataset (i.e., Tiny-ImageNet), the non-robust features are remarkably correlated with the attacked prediction. A brief explanation of the highly correlated example is described in Fig. 4.

heading: 3.4 Bottleneck Controls Information Flow of Robust and Non-robust Features

text:
In this analysis, we will investigate how the bottleneck affects the information flow of the robust and non-robust features and clarify their relation. Recall that the bottleneck refines informative features from the given image samples, and β regulates the total amount of the information that flows into Z. We will compare classification accuracy for the robust and non-robust features along β value and analyze the changes of information flow assigned to each disentangled feature.
In Fig. 5, as β value increases, the accuracy of the robust features is getting higher and decreases after a specific threshold. As theoretically mentioned in 2.1, we can infer that a suitable choice of the β can filter out robust feature units in the intermediate layer. For the excessive β value, we can observe that the bottleneck cannot accurately disentangle adversarial features. For example, the accuracy of the robust and non-robust features are reversed after β = 5.0 in the particular networks of CIFAR-10 and SVHN datasets. In addition, as in Fig. 5(a) and (b), the accuracy of the non-robust features progressively increases. Such results indicate a few robust feature units that are not distilled
from IB are gradually flowing into the non-robust side and dichotomized as non-robust units, thus producing more higher accuracy. It coincides with the analysis of the number of the assigned robust and non-robust channels in Fig. 5(c). We can observe that the number of robust channels constantly diminishes, whereas that of the non-robust channels increases. It indicates the bottleneck delicately weighs the information flow that should be precisely assigned to the robust or non-robust units.
From our analysis section, in conclusion, we have demonstrated the existence of the distilled features using the bottleneck: the robust and non-robust features in the intermediate feature representation. In addition, we have revealed that easily manipulated property of non-robust features is the primary cause of adversarial examples through their high correlation with the adversarial prediction. Based on the fact that the non-robust features break the model prediction, we will suggest an effective way of enhancing the adversarial attack, utilizing the gradient of non-robust features in section 4.

heading: 4 Amplifying Brittleness of Non-Robust Features for Attack

text:
In this section, we intentionally increase the non-robust noise variation σnr to make model classifier fooled based on Eq. (6). Note that increasing σnr induces high erroneous term ∆, thereby going to deviate from the robust prediction. However, this variation σnr is merely an optimizing parameter that cannot be controlled manually. Thus, in order to secondarily have the effect of enlarging σnr, we alternatively utilize the gradients of the non-robust features directly connected with the model
prediction. The gradients of non-robust features in adversarial examples can be described as follows:
Gnr = ∂
∂Z̄nr Lbase(f(X + δ), Y ), (7)
where we define a baseline loss as Lbase(f(X), Y ) = ‖δ‖2 + c · max(max i 6=Y (f(X)i) − f(X)i, 0), instead of cross-entropy loss due to its empirical effectiveness of attack performance [4]. In addition, we use a technique of changes of variables [4] from δ to w for generating an imperceptible yet powerful perturbation, such that δ = 12 (tanh(w) + 1)−X . It serves to smooth out projected gradient descent that clips prematurely to prevent adversarial examples falling into the extreme image domain.
To compute the gradient practically, we firstly calculate ∂ ∂Z̄ Lbase, and multiply it to ∂∂Z̄nr Z̄ by chain rule. Here, the latter gradient equals to non-robust channel index inr, because the intermediate features of adversarial examples can be re-written as: Z̄ = ir · Z̄r + inr · Z̄nr, and the derivative of Z̄ over Z̄nr equals to inr. Thus, the gradients of the non-robust features Gnr can be simplified as inr · ∂∂Z̄Lbase. Using the gradient, we suggest an attack to non-robust features (NRF) by optimizing the following objective:
min δ Lbase(f(X + δ), Y )− ∥∥∥∥inr · ∂∂Z̄Lbase ∥∥∥∥
2
. (8)
In Table 3, NRF shows more effective attack performance than the other standard adversarial attacks in [7, 43, 44], since NRF strengthens the gradients of non-robust features that contains the same effect of increasing σnr to disturb accurate model prediction.
Moreover, we conduct an ablation study on the gradients of robust and non-robust features to probe their influence of prediction changes in Table 4. As expected, maximizing Gnr shows more effective attack performance than minimizing it. It is because maximizing Gnr has an alternative effect of increasing the non-robust noise variation σnr (i.e., large erroneous term ∆ ↑) in Eq. (6). Whereas, manipulating the gradients of robust features Gr shows less attack performance than controlling Gnr, since it cannot directly handle brittle features in the intermediate feature representation. Especially, it seems difficult to break model prediction, while weakening the gradient of robust features, whose noise variation σr have invariance of target prediction even under the adversarial perturbation.

heading: 5 Related Work

text:
Various works have been tried to figure out the reason for adversarial vulnerability in the intermediate feature representation. Inkawhich et al. [45] analyzed how the intermediate features are changed by adversarial attacks and measured layer- and class-wise feature distributions. Engstrom et al. [24] pointed out that there is a shortcoming of DNNs and their embedding, that is, the primary features used in DNNs are contrasting with what human uses. Also, they argued that the robust optimization to learn robust features could address this shortcoming by encoding high-level representations of input data. Jacobsen et al. [46] argued that the reason for adversarial vulnerability lies in invariant characteristics of DNNs to task-relevant features. This invariance makes most regions of input space brittle to adversarial attacks so that the classifiers become relying on a few highly predictive features. In addition, recent works [14, 15] have suggested that the existence of the vulnerability is on the non-robust features, which are inherently included in the data and have unrecognizable properties.
Our work is in line with the concept of the non-robust feature. However, unlike the aforementioned works that directly generated the robust and non-robust datasets to analyze their properties, we reveal that the robust and non-robust features can be completely disentangled in the feature space using Information Bottleneck, and they have semantic information by themselves in fact.

heading: 6 Conclusion and Discussion

text:
Conclusion. To understand where the adversarial brittleness comes from, we have investigated information flow in the intermediate feature space, using Information Bottleneck. By estimating the feature importance of each feature unit based on the added noise variation, we introduce a novel method to explicitly distill the robust and non-robust features in the feature representation. Through extensive analysis of the distilled features, we figure out how the feature units interact with target labels, and corroborate that the non-robust features are highly correlated with the adversarial prediction. In addition, we directly visualize the distilled features in the intermediate feature space and reveal that they have recognizable semantic information by themselves. Based on the properties of the distilled features, we suggest a non-robust feature attack (NRF) utilizing the brittleness in the feature space.
Discussion. The adversarial examples, including our work, can potentially cause negative impacts on various machine learning applications, such as autonomous driving [47] and medical image process [48]. However, by analyzing the nature of robust and non-robust features that can be a primary cause of the adversarial examples, our work contributes important societal impacts in this research field. In this paper, we propose a way of utilizing the gradient of brittle features in the intermediate feature representation (NRF). In future works, we hope to bridge the gap of deploying distilled features into diverse applications such as adversarial detection and defense strategies. Moreover, we would like to note that the last convolutional layer is set to a distilling criterion for Information Bottleneck, since the high-level concepts are included in the higher layer of DNNs. In this sense, it seems a reasonable choice, but how the robust or non-robust features in the lower dimension interact with the much higher dimensional feature units can be another undisclosed future direction to investigate.
Acknowledgments and Disclosure of Funding
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD).

title: Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck