Generalization of Multiplication M-Sequences over Fp and Its Recip- rocal Sequences

Sloane, N.J.A., studied the product or multiplication sequence {zn} on t degrees of the sequence {an} which has the degree of complexity r and gave the answer that the degree of complexity of {zn} can’t be exceeded . Mokayes D. Al Cheikha A. H., studied the construction of the multiplication binary M-Sequences where the multiplication on one M-Sequence or on more than one sequence and gave an illustrated about the question “why the length of the linear equivalent shift register don’t reached the maximum length?” . Al Cheikha A. H., Omar E H.., studied the construction of the multiplication binary M-Sequences with it’s reciprocal sequences, they studied the length of the result multiplication sequence{zn}which is equals [(deg f(x)) deg (f(x))], linear equivalent, properties of the coefficients of the roots of the characteristic polynomial f(x) and it’s reciprocal sequence. Current article and showing the agreement and disagreement between these coefficients and corresponding roots, in the first side, in binary M-Sequences, and in the second side, Mp-Sequences .

is called the initial vector and the characteristic equation of the sequence is; (2) [12][13][14] We are limited in our article to 2 > p .

Definitions and Theorems
Definition 1 The L.F.S.R is a linear feedback shift register if contains only addition circuits and the general term of the sequence {s n } generated through the shift register is the term of the output of the register [3] . Definition 2 The vector ) ,..., , ( , and 1 , ≤ l t R [8][9][10]13] .

Definition 4
The periodic sequence is called an orthogonal set if and only if the set A satisfies the following two conditions; (8) [15,16]

Definition 6
If the function f(x) has the degree n then the reciprocal function of f(x) is the function; and this sequence is called M p -Sequence or M-Sequence over F p [2,[15][16][17] .

Theorem 9
If } { n a is a H.L.R.S in F p and ) (x g is its characteristic prime polynomial of degree k and α is a root of ) (x g in any splitting field of F p then the general term of the sequence } { n a is; And the sequence } { n a has the period 1 − k p [2,18] .
*Our study is limited to the M p -Sequence or M-Sequence over F p and it's period is:

Multiplication Two Reciprocal Mp-Sequences
If f(x) is the characteristic prime polynomial of the M p -Sequence{s n } of degree k in the field F p , and α 1 , α 2 , … α k are it's independent different roots in the F p k , which is a splitting field of F p then the general term of the sequence{s n }is    x x x f which is a prime characteristic polynomial the sequence {s n }, 3 ; F s N n n ∈ ∈ ∀ s n satisfies; We can see the important properties; ( ) ( ) By the same way; We can see the important properties; By the same way; ( ) ( ) The sequence } { n ζ is periodic with the period We can look that one period of the sequence } { n ζ is one period of the sequence {s n }but through reading it by inverse from the right to the left.
The multiplication sequence {z n }, where z n = s n . n ζ is; We can look that one period of the sequence } { n  is one period of the sequence {s n }but through reading it by inverse from the right to the left.
The multiplication sequence {z n }, where z n = s n . n  is; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 2 that is equal , The period of {z n }is 4 and it's the half period of the sequence{s n }and the sequence {z n } is: 0110 0110 0110 0110 …….
We can check that the set of the all periodic permutation of one period is not an orthogonal set for example , for one permutation of the period: 0110 is: 0011 and the sum of the two vectors is 0121 and is not belong to the set of permutations. Figure 3 illustrates the feedback shift registers generating the sequence {z n }; Or; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 2 that is equal to x f x f , The period of {z n }is 4 and it's the half period of the sequence{s n }and the sequence {z n } is: 0110 0110 0110 0110 …….
We can check that the set of the all periodic permutation of one period is not an orthogonal set for example , for one permutation of the period: 0110 is: 0011 and the sum of the two vectors is 0121 and is not belong to the set of permutations. Figure 3 illustrates the feedback shift registers generating the sequence {z n };   Suppose the polynomial which is a prime characteristic polynomial the sequence {a n } as in figure5, 3 ; F a N n n ∈ ∈ ∀ a n satisfies; The general term of the sequence {an}is of the form; 22  3  16  3  3   3  16  3  14  3  2   2  14  3  22  3 And the general solution of the characteristic equation is; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial Is very easy looking that The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial Is very easy looking that We can see the properties; The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial Is very easy looking that And the general solution of the characteristic equation is; The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial Is very easy looking that The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties ;   1  22  3  16  3  3   3  16  3  14  3  2   2  14  3  22  3 And the general solution of the characteristic equation is; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial and it is the characteristic polynomial of the recurring sequence 0 We can see the properties ;   1  25  3  17  3  3   3  17  3  23  3  2   2  23  3  25  3 Is very easy looking that of f(x) and the general term b n is; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; The general term of the sequence {a n }is of the form; Solving the following equations ; We can see the properties ;   1  22  3  16  3  3   3  16  3  14  3  2   2  14  3  22  3 And the general solution of the characteristic equation is; And the sequence is periodic with the period: 3 3 -1 = 26 and the sequence is; 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial and it is the characteristic polynomial of the recurring sequence 0 the roots of g(x) are; We can see the properties ;   1  25  3  17  3  3   3  17  3  23  3  2   2  23  3  25  3 Is very easy looking that 17 23 25 , , corresponding the roots We can see the properties; The reciprocal polynomial of the prime f(x) is the prime polynomial g is the characteristic polynomial of the recurring sequence 2 2 We can see the properties ;   1  25  3  17  3  3   3  17  3  23  3  2   2  23  3  25  3 of f(x) and the general term b n is; Is very easy looking that We can see the properties ;   1  25  3  17  3  3   3  17  3  23  3  2   2  23  3  25  3 of f(x) and the general term b n is; are roots of the characteristic polynomial g(x) corresponding the roots And the sequence is periodic with the period: 3 3 -1 = 26 and the sequen 0 1 1 1 0 0 2 0 2 1 2 2 1 0 2 2 2 0 0 1 0 1 2 1 1 2 0 1 1 1 0 …. (37) The reciprocal polynomial of the prime f(x) is the prime polynomial g is the characteristic polynomial of the recurring sequence 2 2 We can see the properties ;   1  25  3  17  3  3   3  17  3  23  3  2   2  23  3  25  3 Is very easy looking that of f(x) and the general term b n is; We can see the properties; And the general term of the sequence {b n }is; The sequence {b n }is periodic with the period 26 and it is th 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of th reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we hav Thus, the sequence {z n } is a linear nonhomogeneous sequence w homogeneous equivalent is equals 6 that is equal to We can see the properties; And the general term of the sequence {b n }is; The sequence {b n }is periodic with the period 26 and it is the follow 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the seque reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the homogeneous equivalent is equals 6 that is equal to We can see the properties; We can see the properties; And the general term of the sequence {b n }is; The sequence {b n }is periodic with the period 26 and it is th 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of th reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we hav Thus, the sequence {z n } is a linear nonhomogeneous sequence w homogeneous equivalent is equals 6 that is equal to We can see the properties; And the general term of the sequence {b n }is; The sequence {b n }is periodic with the period 26 and it is the fo 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the s reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with homogeneous equivalent is equals 6 that is equal to The sequence {b n }is periodic with the period 3 3 -1=26 and it is the following sequence; 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequence {a n }but through reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; We can see the properties; And the general term of the sequence {b n }is; The sequence {b n }is periodic with the period 26 and it is the follo 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequ reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with th homogeneous equivalent is equals 6 that is equal to x f x f 13, and the sequence {z n } is: 0 1 1 2 0 0 2 0 0 2 1 1 0 0 1 1 2 0 0 2 0 0 2 1 1 0 … Figure 7 illustrates the linear feedback shift registers generating the seque We can see the properties; The sequence {b n }is periodic with the period 26 1 3 3   and it is the following sequence; 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequence {a n }but through reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 6 that is equal to x f x f , The period of {z n }is 13, and the sequence {z n } is: 0 1 1 2 0 0 2 0 0 2 1 1 0 0 1 1 2 0 0 2 0 0 2 1 1 0 ……. Figure 7 illustrates the linear feedback shift registers generating the sequence {z n }; We can see the properties; The sequence {b n }is periodic with the period 26 1 3 3   and it is the follo 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequ reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with th homogeneous equivalent is equals 6 that is equal to x f x f 13, and the sequence {z n } is: 0 1 1 2 0 0 2 0 0 2 1 1 0 0 1 1 2 0 0 2 0 0 2 1 1 0 … Figure 7 illustrates the linear feedback shift registers generating the seque We can see the properties; The sequence {b n }is periodic with the period 26 1 3 3   and it is the following sequence; 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequence {a n }but through reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 6 that is equal to Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 6 that is equal to The sequence {b n }is periodic with the period 26 1 3 3   and it is the following s 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequence { reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the leng homogeneous equivalent is equals 6 that is equal to x f x f , The p 13, and the sequence {z n } is: 0 1 1 2 0 0 2 0 0 2 1 1 0 0 1 1 2 0 0 2 0 0 2 1 1 0 ……. Figure 7 illustrates the linear feedback shift registers generating the sequence {z n The sequence {b n }is periodic with the period 26 1 3 3   and it is the following sequence; 2 1 1 2 1 0 1 0 0 2 2 2 0 1 2 2 1 2 0 2 0 0 1 1 1 0 2 1 1 2 1 0 1 0 0 …. Figure 6 shows register generating {b n }; We can look that one period of the sequence {b n }is one period of the sequence {a n }but through reading it by inverse from the right to the left.
Suppose the multiplication sequence {z n }, where z n = a n .b n , we have; Thus, the sequence {z n } is a linear nonhomogeneous sequence with the length of its linear homogeneous equivalent is equals 6 that is equal to We can see that; 1 ) )(       We can see that; We can check that the set of the all periodic permutation of one period is not an orthogonal set for example, for one permutation of the period: 0 1 1 2 0 0 2 0 0 2 1 1 0 is: 0 0 1 1 2 0 0 2 0 0 2 1 1and the sum of the two vectors is 0 1 2 0 2 0 2 2 0 2 0 2 1 and is don't belong to the set of all permutations of the first period. The linear homogeneous part of z n is LHP(z n ) ={ ' n z }; The characteristic equation of { ' n z } is; We can see that; And the characteristic equation is of the form; 0 1
P5. The length of the linear homogeneous part of the sequence {z n }is equal to 2 ((deg ( )) deg ( )) f x f x − Where the f(x) is the characteristic polynomial of the sequence {S n }.
P6. The period of the sequence {S n } equals the period of the sequence{ n ζ }and equals (p k -1) but the period of the sequence {z n }in the examples1 to 3 is equal to half of this value but in example 4 is equals the quarter of this value. P 7. The set of all cyclic permutations of one period of the multiplication sequence {z n } is not orthogonal set. P 8. There is important difference between the properties of multiplication M-Sequences and M P -Sequences.  furthermore it the permutations between the "coefficients and roots which are in formula of S n " will not be in the formula of n ζ as in the binary M-Sequences.


M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining

Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets. By leveraging the natural suitability of E-commerce, where different modalities capture complementary semantic information, we contribute a large-scale multi-modal pre-training dataset M5Product. The dataset comprises 5 modalities (image, text, table, video, and audio), covers over 6,000 categories and 5,000 attributes, and is 500 larger than the largest publicly available dataset with a similar number of modalities. Furthermore, M5Product contains incomplete modality pairs and noise while also having a long-tailed distribution, resembling most real-world problems. We further propose Self-harmonized ContrAstive LEarning (SCALE), a novel pretraining framework that integrates the different modalities into a unified model through an adaptive feature fusion mechanism, where the importance of each modality is learned directly from the modality embeddings and impacts the inter-modality contrastive learning and masked tasks within a multi-modal transformer model. We evaluate the current multi-modal pre-training state-of-the-art approaches and benchmark their ability to learn from unlabeled data when faced with the large number of modalities in the M5Product dataset. We conduct extensive experiments on four downstream tasks and demonstrate the superiority of our SCALE model, providing insights into the importance of dataset scale and diversity.

Audio Video Figure 1. Our M5Product dataset contains a large variety of modalities (image, text, table, video and audio) that depict the categories, descriptions, materials, properties and purposes of Ecommerce products, and diverse real-world data samples.

Introduction
Self-supervised learning has been driving the rapid development of fields such as computer vision and natural language processing, as well as research on multi-modal representation learning. In particular, it has been shown both from a theoretical [18] and a practical [16,58] perspective that large scale datasets with diverse modalities can effectively enhance the discrimination of generated features and thus improve the performance in vision-language tasks. However, current advances are severely limited by the lack of such large-scale diverse-modality datasets, with the largest public multi-modal datasets only containing text and image modalities and no category information [41].
Given the prevalence of online shopping in daily life, with its natural occurrence of multi-modal information and diverse categories, multi-modal pre-training on Ecommercial products has received increasing attention and led the developments of next-generation technology for several downstream tasks (e.g., multi-modal retrieval, multimodal classification, and clustering). However, even among the present product datasets (e.g., RPC checkout [48], Dress Retrieval [9] and Product1M [55]), the number of categories is insufficient to robustly verify the performance of downstream tasks.
More importantly, the current research community mostly focuses on two modalities (text and image) in both general multi-modal and E-commerce datasets, while ignoring the importance of additional complementary information from structural data as well as video and audio modalities. Tabular data for instance can provide detailed information about properties and characteristics, such as brand, materials, attributes, and scenarios, while audio and video can convey different perspectives, scales, affordances, selling points, characteristics, and use scenarios that are not obvious from images or text alone. The focus on these two modalities is partly due to the lack of datasets with diverse modalities as well as an under-exploration of approaches to balance the modality importance in these settings. In particular, two key challenges are: 1) Modality Interaction: How to learn common representations from unimodal, bimodal, trimodal, and even multi-modal relationships between different modalities using an elegant approach that scales to a large number of modalities; 2) Modality Noise: How to reduce the influence of modality noise (missing and incompleted modalties) during the training process.
To address the problem of insufficient modality diversity and limited scale, while at the same time providing a challenging real-world scenario, we present a very largescale E-commerce multi-modal product dataset M5Product, which is one of the largest and most diverse multi-modal product dataset to date. Our M5Product dataset contains more than 6 million multi-modal samples from 6,232 categories and has more complex and diverse modalities than existing datasets. This allows M5Product to be used for more comprehensive evaluation of the practical application and generalization abilities of multi-modal pretraining models and can improve the modality fusion performance, facilitating new directions in multimodal research. Figure 1 shows the five modalities (image, caption, video, audio, and specification (table)) of our dataset.
To further address the modality fusion limitations of existing methods as well as handle modality noise, we propose a generic framework that takes five-modality data as inputs, as shown in Figure 2. The framework consists of a simple and efficient multi-modal five stream pre-training model named Self-harmonized ContrAstive LEarning (SCALE) and is evaluated on several downstream tasks and compared with several recent state-of-the-art vision-language models [7,27,30,38,42,45,56]. SCALE increases modality alignment effectiveness by implementing a self-harmonized strategy that adapts the alignment weights between different modalities in the contrastive learning modules and masked tasks to adaptively integrate complementary modality information. In summary, our contributions are as follows: • We provide the largest five-modality E-commerce dataset M5Product. Through its large scale, diversity, complex real scenarios and number of modalities, M5Product provides a comprehensive environment for evaluating the generalization performance of multi-modal pre-training models.
• Our Self-harmonized Contrastive Learning (SCALE) framework learns adaptive modality interactions, resulting in more effective modality fusion. We compare SCALE to a comprehensive set of baseline methods and demonstrate its superior performance on the M5Product dataset. •

Related Work
Multi-modal pre-training datasets. Most multi-modal pre-training datasets are collected from social websites (e.g., Twitter and Facebook) and are limited to just two modalities collected for specified tasks. These datasets can be divided into four categories according to their modality composition, i.e., audio/text, video/text, image/text, and others. Among these, LJ Speech [19] and SQuAD [25] are classical audio/text datasets and used for voice synthesis and audio Q&A, while most video/text datasets [2,20,24,32,46,47,51,57] are used for video Q&A. However, these datasets commonly only contain a limited number of samples, limiting their applicability to multi-modal pretraining. Image/text datasets [1,4,8,17,22,23,29,34,41,43,48,53], on the other hand, tend to be larger and have been widely used for pretraining multi-modal models. Among these, the CC 3M [41] with more than three million image-text pairs is the most widely used pre-training dataset, and has recently been expanded to CC 12M [5], the largest text-image cross-  [54] and XMedia [36], where CMU-MOSEI mainly focuses on the emotional analysis and XMedia is utilized for cross-modal retrieval. Aside from the abovementioned datasets, there exist several E-commerce datasets. The Dress Retrieval [9], RPC checkout [48] and Product1M [55] are typical E-commerce multi-modal datasets. The Dress Retrieval dataset contains 20,200 samples from 50 clothing categories, RPC checkout offers 30,000 samples of small retail goods on simple backgrounds and Product1M provides 1.18 million samples from 458 cosmetics classes. Compared with these three datasets, our M5Product is not only larger in terms of categories and data scale, but also contains a more diverse set of modalities. A detailed comparison with other multi-modal pre-training datasets is provided in Table 1.
Within E-commerce, fashion-based tasks have been addressed in among others FashionBERT [13], MAAF [11], Kaleido-BERT [59], M6 [28] and CAPTURE [55]. All existing studies in the E-commerce scenarios focus solely on the image and text modalities and none of the approaches can utilize more modalities. Besides, all existing methods default to assigning the same contribution to different modalities when modeling multi-modal interactions.More  Table   specifically, transformer-based approaches combine highlevel features that are extracted from the different inputs via concatenation, where the uni-modal transformers are trained via masked task constraints or via constructing intermodality losses between different modalities. This restricts the models from effectively prioritizing modalities and tends to limit performance improvements as the number of modalities increases.
Our proposed benchmark fills this gap by exploiting all the diverse modalities of the M5Product dataset and provides a strong baseline for multi-modal pre-training research in the field of E-commerce and beyond.

M5Product Dataset
Data Collections. The dataset is crawled from a popular E-commerce website 2 . and the front page of each Ecommerce product is analyzed to collect the multi-modal information consisting of product images, captions, videos, and specifications (table information) 3 . Duplicate data is removed and audio information is extracted from videos via the moviepy 4 tool and saved in mp3 format. For product specifications, we extract 5,679 product properties and 24,398,673 values to construct a table database coarsely labeled by e-commerce merchants. After processing, the dataset contains 6,313,067 samples. Note, being a realworld dataset, our M5Product is, unlike traditional multimodal datasets, not a complete paired dataset and contains samples with only a subset of modalities as well as longtailed distributed ( Figure 3). We summarize the product characteristics that are relayed by the different modalities in our dataset in Table 2, where APP, USA, SPEC, SELL, PROD, MATE and CATE denote Appearance, Usage, Specification, Selling Point, Production, Material and Category Descriptions, respectively. Quantitative analysis. 1) Diversity: The dataset consists of more than 6,000 classes covering various and massive amounts of E-commerce products such as clothes, cosmetics, and instruments. Figure 1 illustrates the diversity of the modalities and categories and we further provide a description of the data format and the collection process in Section E of the supplementary materials. Finally, a quantita- 2 We are authorised by the company to access and obtain the data. We are further authorised to share the dataset and the detailed license is given in Section A of the supplementary material 3 In this work we focus on core data modalities (image, text, video, audio, and table data) only and do not consider extracted feature representations such as OCR and Motion embeddings that are extracted from core modalities as separate modalities.  Figure 2. An illustration of our M5Product benchmark. It consists of a five-modality E-commerce dataset with a more diverse and complex backgrounds collected from the real-world online-shopping website. It also proposes a SCALE model to capture the maximum modality complementary information for four common downstream tasks: 1) multi-modal retrieval, 2) fine-grained retrieval, 3) multimodal classification, and 4) multi-modal clustering. The benchmark verifies the effectiveness of modality diversity in five widely used modalities.  tive analysis of the category and modality distributions can be found in Section F. Note that about 5% of products are unimodal samples e.g. only either contain images, captions, or tabular properties. 2) Quality: We further provide a comparison between our M5Product dataset and some widelyused datasets for multi-modal pre-training in Table 1. A more extensive comparison with other multi-modal datasets can be found in Section H of the supplementary materials. Compared with existing multi-modal datasets, M5Product is the first extremely large public real-world E-commerce product dataset that contains data of more than two modalities. Moreover, our dataset contains a large amount of instances, i.e., more than six million samples from the 6,232 coarse categories. These abundant data will benefit several downstream tasks such as self-learning, weakly-supervised learning, multi-modal retrieval, cross-modal generation and fine-grained recognition. Additional analysis. In the supplementary materials, we provide dataset collection details in Section B and detail how the dataset is split into training and test in Section D and how annotations are obtained in Section C. We further provide a smaller split, referred to as subset, which is used to show the difference in performance for a smaller dataset. Finally, we provide further insights into the composition of the dataset (missing modalities, unimodal data analysis, and data format) in supplementary Section F.

Our Methodology
As shown in Figure 2, our SCALE framework consists of a self-harmonized contrastive learning module and a selfsupervised multi-modal transformer. In this section, we first provide the architectural design of SCALE in Section 4.1 before describing the five masked tasks that enable the selfsupervised learning of SCALE in Section 4.2. Finally, we present the detailed learning process of SCALE and detail how multi-modal alignment is achieved in Section 4.3.

Architectural Design of SCALE
As depicted in Figure 2, SCALE is a typical singlestream transformer architecture. In the bottom part, the Image/Text/Table/Video/Audio embedding layers and transformers aim to extract modality features and generate token features. Specifically, the text and table encoders are standard transformers to encode the caption and table information of products, respectively. The image encoder instead takes proposals extracted by bottom-up-attention [3] as inputs, while ordinal frames sampled from the video are fed into the video encoder. For the audio encoder, SCALE extracts MFCC [33] features from audio. After being processed by the separate modality encoders, the token features of different modalities are concatenated and fed into a Joint Co-Transformer (JCT) module to capture the token relationships between different modalities. Missing Modalities. Zero imputation of missing modalities is leveraged to utilize all available data when training SCALE. We provide experimental evidence that SCALE benefits from the incomplete samples in Section I of the supplementary material.

SCALE by Masked Multi-Modal Tasks
Similar to previous works, we utilize several pretext tasks (PRE) to facilitate self-supervised learning of SCALE in the Joint Co-Transformer module. For modalitywise feature learning from the image and text modalities, we adopt the Masked Region Prediction task (MRP) and the Masked Language Modeling task (MLM), respectively, after the JCT. Utilizing the characteristics of the table, video, and audio modalities, we further propose a Mask Entity Modeling task (MEM), Mask Frame Prediction task (MFP), and Mask Audio Modeling task (MAM) following a similar strategy of predicting masked tokens. In all masked tasks, the ground-truth labels are the features of masked areas. For all masking tasks, 15% of the inputs are masked out and the remaining inputs are used to reconstruct the masked information. Please note that unlike in the MLM task, where 15% of individual words are masked, 15% of the entities (properties, brand names, etc.) are entirely masked out for the MEM task. This drives our model to learn better table representations to recover the masked inputs, which is illustrated in Section 5.3. The loss function of the ith modality is defined as: where t ¬msk denotes the unmasked tokens surrounding the masked token t msk , θ represents the network parameters, and M i and M ¬i are the ith modality and the remaining modalities, respectively.

Self-harmonized Inter-Modality Contrastive Learning
Self-harmonized Inter-Modality Contrastive Learning (SIMCL) is at the core of our proposed SCALE framework. It aims to facilitate the semantic alignment between different modalities via a self-harmonized strategy for adaptive Inter-Modality Contrastive Learning (IMCL). For a minibatch of modality samples D ∈ R B×M ×F , where B, M and F denote the batch size, number of modalities, and embedding dimension, respectively, we first construct the contrastive loss between each modality.

Inter-Modality Contrastive Learning
Inter-Modality Scores where each sample has two modalities (0) and (1) with the remaining N -1 samples from the other modality, resulting in 2(N -1) negative pairs. For a modality pair (d i ), the cross-modal contrastive loss of each modality pair is: where sim is the cosine similarity, τ is the temperature parameter and 1 [k =i] is a binary indicator function, and 1=1 for k = i and 0 otherwise.
In most prior work, only two modalities are considered and Eq. 2 can be used. However, when considering trimodal data or data with even more than three modalities, it is not suitable to directly fit the loss function as it does not account for the difference in complementary information that different modalities contribute. To solve this problem, we define a simple but effective self-harmonized method to model the complementary process of the inter-modal relationships. We introduce a modality alignment score matrix, to encode the relationships among the inter-modal losses L CL and the intra-modal losses L Mi . The alignment score matrix S for each data sample is initialized by a zero matrix and updated as free model parameters. To obtain modality importance scores for each modality combination, we apply the softmax function to S. Finally, the importance scores are multiplied to generate the modality alignment score S as S = S · softmax(S). The learning process is shown in Figure 4 and illustrates that SIMCL takes full advantage of the inter-modal relationships. Given the modality alignment score S, the triangular part S is selected to weight the inter-modal losses L CL and the diagonal part S \ is uti-lized to constrain the intra-modal losses L Mi , resulting in the weighted loss: where logit is the the loss logit.

Experiments
Implementation Details. We use BERT [10] to initialize the text transformer of our proposed SCALE framework, while the remaining transformers are randomly initialized. Both the single-modality encoders and the multi-modal fusion encoders consist of 6 transformer layers each, adding up to a total of 12 transformer layers. The hidden state size of each modality transformer is 768 and the maximum sequence length for the captions and tables are set to 36 and 64, respectively. Using the same setting as in [30] 5 , we utilize Faster R-CNN [40] with a backbone ResNet101 [15] pre-trained on the Visual Genome dataset [23] to extract region features of selected 10 to 36 bounding boxes with high-class detection probability for each image. We train SCALE with a total batch size of 64 for 5 epochs using the Adam optimizer [21] with a warm-up learning rate of 1e-4. Additional implementation details of our model are provided in Section G of the supplementary material. Baselines. We compare SCALE to the following eight alternative pre-training methods that utilize image and text modalities as well as combinations of both: Bert [10] (Text based ), Image based , ViLBERT [30], CLIP [38], VL-BERT [42], VisualBERT [27], UNITER [7] and CAP-TURE [56]. Image based and BERT [10] are 12-layer transformers based on the MLM (Mask Language Modeling) or MRP (Mask Region Prediction) task using image or text modality, providing single-modal baselines for the product retrieval, classification, and clustering tasks. To ensure a fair comparison, the same hidden size of 768 is chosen for all baselines. Evaluation. We consider the following four downstream tasks to evaluate the learned representations: 1) Multimodal retrieval: This task aims to find the most relevant target products using combinations of two or more modalities. A pair is considered a match if both belong to the same category; 2) Fine-grained multi-modal retrieval: Retrieval on an instance level, where only samples of the same product (i.e. color, model, shape, and style) are considered a match 6 ; 3) Multi-modal classification: Product category classification given the multi-modal features extracted from the joint co-transformer of SCALE using a linear classifier; and 4) Multi-modal clustering: Product category clustering using k-Means clustering and the same features as in the classification setting. For product retrieval, we adopt the widely used metrics mean Average Precision (mAP) and Precision (Prec) [14,31,49] to evaluate the retrieval accuracy on the two retrieval tasks. For product classification and clustering, all methods are evaluated using the Classification Precision (Classification accuracy), Normalized Mutual Information (NMI) [52] and Purity metrics. In all experiments, models are trained on the training split. The pretrained model is then applied to extract the modality features of the gallery and test splits for the product retrieval and clustering tasks. For the classification task, we finetune the pre-trained model on the classification subset containing 1,805 categories/classes and utilize the finetuned model to extract the features of the classification test set.

Modality Diversity
To examine the performance of our proposed SCALE framework and to verify the benefits of diverse modalities and dataset scale, we train SCALE with an increasing number of modalities and observe the variations in classification and multi-modal retrieval performance both for the whole M5Product dataset and the subset. More specifically, fused features are extracted from the joint co-transformer (JCT) of our SCALE after finetuning for the classification task and after pre-training and finetuning for the (coarse) multimodal retrieval task. Results in Table 3 show that performance increases across all settings as modalities are added, illustrating the benefit of complementary modality information to learning multi-modality feature representations. It can also be observed that modality gains are larger on the whole dataset, supporting Interesting Observation 1.
We further provide results for an extensive set of modality combinations to verify SCALEs effectiveness in leveraging the diverse modalities of our M5Product dataset. Table 4 provides results for the coarse-and fine-grained multimodal retrieval tasks as well as the classification task after finetuning the model. As in the previous experiment, noticeable improvements can be observed as additional modalities are added. In particular, the addition of the text modality leads to high modality gains, verifying the benefits of including more diverse modalities that can capture different views of the same product. Interestingly, performance on the coarse-grained retrieval task is significantly worse than on the fine-grained retrieval task in most cases, indicating the complexity of the M5Product dataset and the diversity of the products in each category. Semantic Alignment. To additionally demonstrate the importance of modality diversity, we compute the modality correlation, the average cosine similarity between image and text features as obtained by the JCT, for an increasing number of modalities. Figure 5 illustrates that the semantic alignment capability of the pre-training model increases as Table 3. The (pretrain/finetune) performance gains from sequentially adding more modalities using SCALE on the subset (top) and the whole dataset (bottom). The retrieval performances are based on the features extracted from pretrain and finetune stages.   the number of modalities grows.

Multi-modal Downstream Tasks
We evaluate SCALE on the M5Product dataset for the product retrieval, classification, and clustering tasks and compare results to several benchmark approaches in Table 5. For the Image based and BERT [10] models, which only utilize the image and text features, respectively, the extracted features are fed directly into the classification model. For our SCALE approach, we utilize the fused modality features generated by the joint co-transformer, pre-trained on both image and text modalities. Only utilizing the image and text modalities allows us to facilitate a fair comparison to the recent state-of-the-art approaches ViLBERT [30],  CLIP [38], VL-BERT [42], VisualBERT [27], UNITER [7] and CAPTURE [56]. Comparing our SCALE framework to the unimodal models, Image based and Bert [10], we observe that exploiting multi-modal data significantly improves the performance across all tasks. We further observe that SCALE, by leveraging SIMCL, can efficiently fuse the modalities and outperform all the baseline approaches (Interesting Observation 2).

Ablation Studys and Visualization
To explore how SIMCL influences IMCL and the Pretext tasks, we conduct several ablation studies. Table 6 illustrates that improvements of approximately 2% are obtained in the classification task and more than 2% for the coarse-grained retrieval task when including both, highlighting the importance of both the Pretext tasks and the effective modality fusion of SIMCL. We further analyze the effect of the MEM pretext task for the table modality and show the benefit of masking out complete entities over masking out individual tokens (MLM) in Table 7. This benefit can be attributed to the fact that MEM ensures that SCALE learns representations that encode the semantic information of complete entities. Finally, we evaluate the performance of modelling the text and the table modalities using individual modality encoders and compare SCALEs retrieval performance to a baseline where text and table information is concatenated and fed to a single transformer, resembling the process of BERT [10]. By modelling both modalities individually, results in Table 8 illustrate that more information can be preserved and we hypothesize that using a single transformer leads to a loss in table modality information for the benefit of the more expressive text modality. Figure 6 shows t-SNE visualizations of the extracted features for the JCT module of our SCALE model and the alternative approach ViLBert [30] on the M5Product dataset. SCALE not only better distinguishes different classes but also improves class compactness compared to the ViLBert model. Further, the attention attribution for different modalities are shown in Figure 7 and verify that the visual features generated by SCALE are object-oriented and semantically interpretable.

Limitations and future work
The experimental evaluation showed that SCALE is able to learn efficient representations from a large number of modalities for retrieval, classification, and clustering. However, more evaluation of the generative capabilities of the models representations is lacking and tasks such as image and caption generation could be promising directions to explore. We further provide some of SCALEs failure cases in supplementary Section J. Potential negative societal impact. As a result of the strict ethical considerations used in the data collection process, where among others personally identifiable information has been removed, M5Product does not pose any ethical risks.

Conclusion and Discussion
To facilitate multi-modal pre-training, we present the M5Product dataset, which is the largest available multimodal E-commerce product dataset, consisting of five core modalities (image, text, table, video, and audio). To further promote multi-modal research in retail and increase seller and buyer engagement, we also propose the novel SCALE multi-modal pre-training framework. By utilizing Self-harmonized Inter-Modality Contrastive Learning (SIMCL), SCALE is able to model and exploit modality relationships effectively and outperforms previous approaches on the M5Product multi-modal retrieval, classification, and clustering tasks. We believe that both the dataset and the proposed framework work will inspire research on scaling multi-modal pre-training beyond the commonly used image and text modalities.

A. Dataset License
Our M5Product dataset is released under CC BY-NC-SA 4.0 license and can freely be used for noncommercial purposes. More detailed information can be found at https://xiaodongsuper.github. io/M5Product_dataset/terms_of_use.html, which also provides dataset details and usage guidance. Note: For anonymity reasons, the link is not included during the review process.

B. Annotation Collection
We resort to crowd-sourcing to obtain human annotations for the product retrieval task. Specifically, we present human annotators with a matching task, where annotators are asked to select the matching image-text pairs for a given query image-text pair. In our crowdsourcing system, each matching task is presented to five different human annotators and a typical example of our interface is shown in Figure 8. The left part of the interface shows the current query data (image and text), while the right side depicts an example from the candidate list. The annotators are then asked to choose from two options: mismatched and uncertain. The default labelling option is matched. The interface also displays the number of examples that have been reviewed and the total amount of examples to review. Each annotation task, can be considered as a binary classification task for the human worker, where he/she has to decide if the pair is a match or not. For each estimated task, the annotators receive a payment of 3 cents RMB.

C. Annotation
The retrieval task annotation for any query sample consists of all the matched instances in the gallery split. To construct a reliable gallery set, we first use a ResNet50 [15] and Bert-Base [10] to extract features and construct the query candidate pool from all the data that is not contained in the training subset. Specifically, we sample an instance from a category that contains more than 2,000 instances and extract the image and text features. We then concatenate the features and compute the cosine similarity to all other instances of the dataset to produce a pre-ranked candidate list in order to minimize the labelling cost. The final size of the candidate shortlist for each query is 500, which is about 0.01% of the whole gallery split. During the crowd-sourced annotation process, human workers review both images and captions in the candidate list to select which samples are matched with the query instance.
Annotation Rules. It is quite challenging to define whether two images contain the same product when critical aspects are not given in their captions and images. In our annotations, we use product images and their captions as the primary materials for gallery construction. Hence, we define several rules to determine the "same product" condition and provide them as instructions to the annotators. Images contain the same product, if: 1. The two images are in different conditions (e.g., backgrounds, angles, etc), but the products in both images are the same.
2. They should have the same color/model/shape/style, or other features that can be distinguished by humans.
3. The caption has the same product name but the product description differs.
4. They share more than one characteristic such as appearances, materials, colors and so on.
To ensure labeling consistency, each annotation pair is labeled by five human workers in the crowd-sourced platform. In the process, we first make a small dataset from our query list as a Gold Problem to evaluate the annotation capability of each human worker. Based on the labeled results ("Matched" or "Not Matched") from human workers and their annotation capability, we utilize the weighted GLAD [50] inference algorithm to determine the final accepted labels.

D. Dataset Split
The M5Product dataset is split into several parts to ensure consistent training and evaluation of the models on the various tasks. The training set contains 4,423,160 samples from 3,593 classes.
Retrieval To evaluate models on the retrieval tasks, the remaining data is split into gallery-c and query-c sets, which are used for the coarse-grained retrieval task, and gallery-fg and query-fg sets, which are used for the fine-grained retrieval task. The difference between the two retrieval tasks lies in the granularity of their annotation. In the fine-grained task, only identical products are considered a match (for example, all IPHONE 11 Black), while in the coarse-grained task, category labels are being used to group products from each category (for example, all phones are considered a match).
To construct the fine-grained sets, we extracted all cosmetics categories and, using the abovementioned annotation procedure, finally obtained 1,991 query-fg samples and 117,858 gallery-fg samples. The query-c and gallery-c sets contain 24,410 and 1,197,905 samples, respectively. Among the samples in the gallery-c set, 249,614 samples are matched with samples in the query-c set, while 948,291 samples do not match. These unmatched samples are added to the gallery-c set to increase the difficulty of the retrieval task. We further report the finetuned retrieval performance  Figure 8. UI for huamn annotation on product retrieval task.
in the paper, which corresponds to retrieval performance after finetuning the model using the classification training set (see next paragaraph).
Finetuning, Classification and Clustering For the classification and clustering tasks, we sampled 1,805 categories from the whole dataset and obtained 18,526 train samples and 4,632 test samples. We first finetune our SCALE using the classification training set and then extract features from the finetuned model to perform the classification and clustering tasks.

E. Data Format
The dataset consists of 6,313,067 products uploaded by 1,000,517 merchants, where merchant information has been removed to ensure anonymity. In the following, we outline the different modalities: Image data Each product has at least five product images, where the first image is the main image that gives the detailed overview of the product, while the rest depict its functionalities or characteristics. We pick all the main images to construct the dataset. Caption/text data are provided by the 1,000,517 merchants. Note that the text description does not always match well with the other modalities. Video data are used to showcase the products' usage and characteristics to customers. In our dataset, these videos are recorded at a speed of 24 frames per second (FPS). To reduce the amount of redundant information that is contained in adjacent frames and the dataset as a whole, we only select one frame per second. Audio data are extracted from the video data. We extract the corresponding audio information of all sampled video frames. Then the audio frames are transformed into spectrograms using Mel-Frequency Cepstral Coefficients (MFCC) [33]. We set the frame size and hop size as 1,024 and 256, respectively.
Tabular data are a special kind of database that records some additional product characteristics such as appearance, purpose and producer. The tabular data is indexed by the product ID and collected from the whole product database. There are 5,679 different types of property information and 24,398,673 unique values.
F. Unimodal and unpair analysis 1) Unimodal analysis: Figure 9 gives the video, text and merchant distributions. From the figure, we can find that the video duration, the text length and the merchant number range from 1 to 60 seconds, 20 to 40 words and 1 to 10 product numbers, respectively. This variation further illustrates the real-world nature of our dataset. 2) Unpair analysis: In the data collection process, there are 82,577 invalid URLs for the image modalities (1.3% of the products), while the number of samples that contain both the Image and Text modalities is 6,230,490. Further taking into account the table modality, the number of complete samples drops by 1.4% to 6,225,598 samples that have all three modalities. Overall, the dataset contains 5,050,078 samples that contain all five modalities. This means that about 20% of the samples are incomplete. This is mostly due to merchants being biased towards specific modalities, which is a common scenario in the real world.

G. Implementation Details
Our models are implemented in Pytorch [35]. To speed up training, we use Nvidia Apex 8 for mixed precision training. All models are trained on 4 Nvidia 3090 and 2080ti GPUs on our workstations. We use Adam [21] to optimize the parameters of our model, with an initial learning rate of 1e-4, and use a linear learning rate decay schedule with a temperature parameter of 0.1.

H. Dataset Comparison
A comprehensive comparison between our M5Product dataset and other widely used multi-modal pre-training datasets is shown in Table 9. From the table, we can observe that our M5Product not only has more diverse modalities but also contains a large amount of data samples from an abundant amount of categories.

I. Missing data verification
Results in Table 10 show the superiority of our methods over the standard approach of ignoring incomplete samples. We compare two variants of our SCALE framework: 1) SCALE (full-modality) and our proposed SCALE. The only difference between the two methods is the input. The input of the former only includes complete samples (all modalities present), while the input of the latter includes the incomplete modality samples. The verification is performed on the subset dataset as mentioned in the main article. Figures 10, 11, and 12. The first column represents the image and text modality of the query sample, while the eight images to its right belong to the matched results from the gallery set. In the matched results, the samples boxed in blue are the correctly matched samples, while the samples boxed in red are mismatched. These retrieval results illustrate that the learned embeddings are discriminative. However, in a few cases, the recalled samples are not matched due to the limited number of category samples in the gallery set or similar descriptions in the text data.

K. More Visualization
Additional attention visualization results are provided in Figures 13 and 14. Similar to the illustrations in the main paper, these visualizations show that SCALE can learn the detailed semantics in the images and the text.

L. Code and Our dataset.
The code is provided in the supplementary and we also include a few full-modality examples of the dataset. Due to space constraints, we have refrained from sharing the whole dataset.
Car silk circle foot mats Changan Ease cs35 new cx20 Yuexiang V5 to Shang XT Ben Ben mini main driver single piece Zhang Xiaoquan kitchen knife home stainless steel slicing knife chef special kitchen knife cutting vegetables and meat free grinding kitchen knives lulu yoga pants female outer wear net red nude sense fitness pants high waist lifting hip running nine points tight original sports pants Old daddy shoes with rainbow mesh shoes women 2020 summer new thick bottom muffin breathable mesh surface inside high sports shoes TOMY Domeka alloy car model male toys TOMICA Ferrari 246GT / F40 / 512BB / 365GT Yun Yun inter Anji white tea 2020 new tea Ming Qian first pick authentic master spring tea premium rare bulk 100g


Improving Hierarchical Adversarial Robustness of Deep Neural Networks

Do all adversarial examples have the same consequences? An autonomous driving system misclassifying a pedestrian as a car may induce a far more dangerous -- and even potentially lethal -- behavior than, for instance, a car as a bus. In order to better tackle this important problematic, we introduce the concept of hierarchical adversarial robustness. Given a dataset whose classes can be grouped into coarse-level labels, we define hierarchical adversarial examples as the ones leading to a misclassification at the coarse level. To improve the resistance of neural networks to hierarchical attacks, we introduce a hierarchical adversarially robust (HAR) network design that decomposes a single classification task into one coarse and multiple fine classification tasks, before being specifically trained by adversarial defense techniques. As an alternative to an end-to-end learning approach, we show that HAR significantly improves the robustness of the network against $\ell_2$ and $\ell_{\infty}$ bounded hierarchical attacks on the CIFAR-10 and CIFAR-100 dataset.

Introduction
Deep neural networks (DNNs) are highly vulnerable to attacks based on small modification of the input to the network at test time (Szegedy et al., 2014). Those adversarial perturbations are carefully crafted in a way that they are imperceptible to human observers, but when added to clean images, can severely degrade the accuracy of the neural network classifier. Since their discovery, there has been a vast literature proposing various attack and defence techniques for the adversarial settings (Szegedy et al., 2014;Goodfellow et al., 2015;Kurakin et al., 2017;Madry et al., 2018;Wong et al., 2019a). These methods constitute important first steps in studying adversarial robustness of neural networks. However, there exists a fundamental flaw in the way we assess a defence or an attack mechanism. That is, we overly generalize the mistakes caused by attacks. Particularly, the current approaches focuses on the scenario where different mistakes caused by the attacks are treated equally. We argue that some context do not allow mistakes to be considered equal. In CIFAR-100 (Krizhevsky, 2009), it is less problematic to misclassify a pine tree as a oak tree than a fish as a truck.
As such, we are motivated to propose the concept of hierarchical adversarial robustness to capture this notion. Given a dataset whose classes can be grouped into coarse labels, we define hierarchical adversarial examples as the ones leading to a misclassification at the coarse level; and we present a variant of the projected gradient descent (PGD) adversaries (Madry et al., 2018) to find hierarchical adversarial examples. Finally, we introduce a simple and principled hierarchical adversarially robust (HAR) network which decomposes the end-toend learning task into one coarse and multiple fine classification tasks, before being trained by adversarial defence techniques. Our contributions are • Motivated by the empirical observation that untargeted PGD attacks yield misclassifications within the same coarse label (Section 2.1), we present a worst-case targeted PGD attack to find hierarchical adversarial examples. The attack iterates through all candidate fine labels until a successful misclassification into the desired target (Section 2.2).
• We propose a novel architectural approach, HAR network, for improving the hierarchical adversarial robustness of deep neural networks (Section 3). We empirically show that HAR networks significantly improve the hierarchical adversarial robustness against ∞ attacks ( = 8 255 ) and 2 attacks ( = 0.5) on CIFAR-10 and CIFAR-100 (Section 4 and Appendices D, E).
• We benchmark using untargeted PGD attacks as well as the proposed worst-case targeted PGD attack. In particular, we include an extensive empirical study on the improved hierarchical robustness of HAR by evaluating against attacks with different p norms, varying PGD iterations and . The result shows that the proposed worst-case targeted attack provides a more accurate empirical representation of the hierarchical adversarial robustness of the model (Section 4.2).
• We show that the iterative targeted attack formulated based on the coarse network are weaker hierarchical adversarial examples compared to the ones generated using the entire HAR network (Section 4.3).

Hierarchical Adversarial Examples
The advancement in DNN image classifiers is accompanied by the increasing complexity of the network design (Szegedy et al., 2017;He et al., 2016), and those intricate networks has provided state-of-the-art results on many benchmark tasks (Deng et al., 2009;Geiger et al., 2013;Cordts et al., 2016;Everingham et al., 2015). Unfortunately, the discovery of adversarial examples has revealed that neural networks are extremely vulnerable to maliciously perturbed inputs at test time (Szegedy et al., 2014). This makes it difficult to apply DNN-based techniques in mission-critical and safety-critical areas.
Another important development along with the advancement of DNN is the growing complexity of the dataset, both in size and in number of classes: i.e. from the 10-class MNIST dataset to the 1000-class ImageNet dataset. As the complexity of the dataset increases exponentially, dataset can often be divided into several coarse classes where each coarse class consists of multiple fine classes. In this paper, we use the term label and class interchangeably.
The concept of which an input image is first classified into coarse labels and then into fine labels are referred to as hierarchical classification (Tousch et al., 2012). Intuitively, the visual separability between groups of fine labels can be highly uneven within a given dataset, and thus some coarse labels are more difficult to distinguish than others. This motivates the use of more dedicated classifiers for specific groups of classes, allowing the coarse labels to provide information on similarities between the fine labels at an intermediate stage. The class hierarchy can be formed in different ways, and it can be learned strategically for optimal performance of the downstream task (Deng et al., 2011). Note that it is also a valid strategy to create a customized class hierarchy and thus be able to deal with sensitive missclassification. To illustrate our work, we use the predefined class hierarchy of the CIFAR-10 and the CIFAR-100 dataset (Krizhevsky, 2009): fine labels are grouped into coarse labels by semantic similarities.
All prior work on adversarial examples for neural networks, regardless of defences or attacks, focuses on the scenario where all misclassifications are considered equally (Szegedy et al., 2014;Goodfellow et al., 2015;Kurakin et al., 2017;Madry et al., 2018;Wong et al., 2019a). However, in practice, this notion overly generalizes the damage caused by different types of attacks. For example, in an autonomous driving system, confusing a perturbed image of a traffic sign as a pedestrian should not be treated the same way as confusing a bus as a pickup truck. The former raises a major security threat for practical machine learning applications, whereas the latter has very little impact to the underlying task. Moreover, misclassification across different coarse labels poses potential ethical concerns when the dataset involves sensitive features such as different ethnicities, genders, people with disabilities and age groups.
Mistakes across coarse classes lead to much more severe consequences compared to mistakes within coarse classes. As such, to capture this different notion of attacks, we propose the term hierarchical adversarial examples. They are a specific case of adversarial examples where the resulting misclassification occurs between fine labels that come from different coarse labels.
Here, we provide a clear definition of the hierarchical adversarial examples to differentiate it from the standard adversarial examples. We begin with the notation for the classifier. Consider a neural network F (x) : R d → R c with a softmax as its last layer (Hastie et al., 2009), where d and c denote the input dimension and the number of classes, respectively. The prediction is given by arg max i F (x) i .
In the hierarchical classification framework, the classes are categorized (e.g. by the user) into fine classes and coarse classes 1 . The dataset consists of image and fine label pairs: {x, y} n . In the later, we use the set theory symbol ∈ to characterize the relationship between a fine and a coarse label: y ∈ z if the fine label y is part of the coarse class z. Note that this relation holds for both disjoint and overlapping coarse classes. Given an input data x, suppose its true coarse and fine labels are z * and y * respectively. Under the setting defined above, a hierarchical adversarial example must satisfy all the following properties: • the unperturbed data x is correctly classified by the classifier: arg max i F (x) i = y * ; • the perturbed data x = x + δ is perceptually indistinguishable from the original data x; • the perturbed data x is classified incorrectly: arg max i F (x ) i = y where y = y * ; • the misclassified label belongs to a different coarse class: y ∈ z * .
Notice that satisfying the first three properties is sufficient to define a standard adversarial examples, and that hierarchical adversarial examples are special cases of adversarial examples. It is worth mentioning that measuring perceptual distance can be difficult (Li et al., 2003), thus the second property is often replaced by limiting that the adversary can only modify any input x to x + δ with δ ∈ ∆. Commonly used constraints are -balls w.r.t.
p -norms, though other constraint sets have been used too (Wong et al., 2019b). In this work, we focus on ∞ -and 2 -norm attacks.

Empirical Observations on Untargeted Attacks
A common class of attack techniques are gradient-based attacks, such as FGSM (Goodfellow et al., 2015), BIM (Kurakin et al., 2017) and PGD (Madry et al., 2018), that utilize gradient (first-order) information of the network to compute perturbations. Such methods are motivated by linearizing the loss function and solving for the perturbation that optimizes the loss subject to the p -norm constraint. Their popularity is largely due to its simplicity, because the optimization objective can be accomplished in closed form at the cost of one back-propagation. In the context of our work, an important notion is that the attack can be either targeted or untargeted. The untargeted attack aims to perturb the image in a way that the altered image will be classified incorrectly. On the other hand, the goal of the targeted attack is to cause misclassification into a specified target class.
Despite a plethora of attack techniques has been proposed, PGD remains as a fundamental algorithm for many intricate attacks and a standard empirical approach for measuring model robustness (Gowal et al., 2019;Mao et al., 2021;Tashiro et al., 2020;Croce and Hein, 2020). For instance, the recent SOTA method AutoAttack (Croce and Hein, 2020) consists an ensemble of powerful attacks where PGD is directly used for evaluating model's robustness against untargeted adversaries.
To motivate the hierarchical adversarial perturbations in Section 2.2, we start with an empirical analysis on the mistakes made by networks on clean and perturbed images. We found that untargeted ∞ -norm PGD20 attacks ( = 8 255 ) are more likely to yield misclassifications within the same coarse label, captured by the results shown in Table 1 where we compute the percentage of misclassified fine labels that are still correctly classified at the coarse level: # still classified correctly at the coarse level # misclassified at the fine level × 100%. The two robustified models are trained using PGD10 adversarial training (ADV) Madry et al. (2018) and TRADES Zhang et al. (2019). We make two important observations from the results in Table 1. First, note Table 1: Percentage of the misclassified fine labels that are still correctly classified at the coarse level, e.g., misclassifying a ship as an airplane which is also in the"vehicle" coarse label of CIFAR-10. that misclassifications from the untargeted attack are more likely to be within the same coarse label. This is compared to the chance of being classified as a random coarse label, which is 50% and 5% for CIFAR-10 and CIFAR-100 respectively. Second, notice that given an image that is misclassified at the fine level, whether perturbed or not does not significantly change its likelihood of being misclassified at the coarse level. For example, we observe that for a TRADES-robustified model trained on CIFAR-10, around 77% of both misclassified clean images and PGD-perturbed images are still correctly classified at the coarse level. This suggests that in terms of degrading hierarchical robustness, an untargeted PGDperturbed input is just as ineffective as an unperturbed one. This motivates us to adopt a new approach for generating hierarchical adversarial perturbations, as well as a metric for robustness evaluation.

Generating Hierarchical Adversarial Perturbations
The main idea of gradient-based attacks can be summarized as follows. Given the prediction of F (x) and a target label y, the loss function of the model is denoted by (x, y) (F (x), y), e.g., a cross-entropy loss. Here, we omit the network parameter w in the loss because it is assumed to be fixed while generating adversarial perturbations. Note that the choice of y and whether to maximize or minimize the loss depend on if the attack is targeted or untargeted. For a targeted ∞ attack, gradient-based methods rely on the opposite direction of the loss gradient, − sign ∇ x (x, y), to solve for the perturbation that minimizes the loss with respect to a non-true target label (y = y * ). Despite its simplicity, gradient-based attacks are highly effective at finding p -bounded perturbations that lead to misclassifications.
In our work, we introduce a simple variant of the PGD adversary to find hierarchical adversarial examples. Given an input image with true coarse and fine labels z * and y * respectively. Let x j denote the perturbed input at iteration j, we define: where the target labelŷ comes from a different coarse class:ŷ ∈ z * . Algorithm 1 summarize the procedures for generating an ∞ -constrained hierarchical adversarial examples. The projection operator Π after each iteration ensures that the perturbation is in an -neighbourhood of the original image. We also adopt the random initialization in PGD attacks (Madry et al., 2018): Algorithm 1: A worst-case approach for generating ∞ -bounded hierarchical adversarial example based on a targeted PGD attack. Input: A pair of input data (x, y * ), where fine label y * belongs to the coarse label z * ; 1 a neural network F (·); loss function (·); ∞ constraint of ; number of PGD iterations k; PGD step-size α.
Define: S = {y | y ∈ z * }, a collection of all fine labels that do not belong in the There are several approaches to choose the target class (Carlini and Wagner, 2017). The target class can be chosen in an average-case approach where the class is selected uniformly at random among all eligible labels. Alternatively, they can be chosen in a strategic way, a best-case attack, to find the target class which requires the least number of PGD iterations for misclassifications. In our work, we consider a worst-case attack by iterating through all candidate target labels, i.e., fine labels that do not belong in the same coarse class. This iterative targeted attack process terminates under two conditions: 1. perturbation results in a successful targeted misclassification; 2. all candidate fine labels have been used as targets.

Related Work on Hierarchical Classification
In image classification domain, there is a sizable body of work exploiting class hierarchy of the dataset (Tousch et al., 2012). For classification with a large number of classes, it is a common technique to divide the end-to-end learning task into multiple classifications based on the semantic hierarchy of the labels (Marsza lek and Schmid, 2008; Liu et al., 2013;Deng et al., 2012). They are motivated by the intuition that some coarse labels are more difficult to distinguish than others, and specific category of classes requires more dedicated classifiers. A popular hierarchy structure is to divide the fine labels into a label tree with root nodes and leaf nodes, and Deng et al. (2011) propose an efficient technique to simultaneously determine the structure of the tree as well as learning the classifier for each node in the tree. It is also common to use the predefined hierarchy of the dataset (Deng et al., 2012).

Hierarchical Adversarially Robust (HAR) Network
To improve the hierarchical adversarial robustness of neural networks, we propose a simple and principled hierarchical adversarially robust (HAR) network which decompose the endto-end robust learning task into two parts. First, we initialize a neural network for the coarse classification task, along with multiple networks for the fine classification tasks. Next, all the networks are trained using adversarial defence techniques to improve the robustness of their task at hand. The final probability distribution of all the fine classes are computed based on Bayes Theorem. For brevity, we use coarse neural network (CNN) and fine neural network (FNN) to denote the two different types of networks. Intuitively, the HAR network design benefits from a single robustified CNN with improved robustness between coarse classes, and multiple robustified FNN with improved the robustness between visually similar fine classes.

Architecture Design of HAR
Instead of the traditional flat design of the neural network, HAR consists of one CNN for the coarse labels and several FNNs for the fine labels. Note that there is a one-to-one correspondence between a particular FNN and a specific group of fine labels. Such a module design mimics the hierarchical structure of the dataset where the fine classes are grouped into coarse classes. Recall that our definition of the neural network F (x) includes the softmax function as its last layer, so the output of the network can be interpreted as the probability distribution of the classes: P (y | x). Conditioned on the probability of the coarse class, we can define the fine class probability as Here, the probability distribution of the fine classes are computed as the product of two terms. Given an input x, the first term P (y | x, z) represents the probability of x being a fine label y where y belongs to a coarse class z. This is essentially FNNs' prediction of the fine classes within a coarse category. The second term P (z | x) represents the probability of x being a coarse label z, and it can be understood as the prediction from the CNN. With this decomposition of the original learning task, we can reconstruct the fine label distribution by probabilistically combining the predictions from the different networks.
An important advantage of this flexible modular network design is that it allows us to train the component networks using adversarial defence techniques to improve the robustness of their associated task. Especially, a robustified coarse neural network leads to improved hierarchical adversarial robustness between coarse labels. During training, each component of the HAR network is trained independently, allowing them to be trained in parallel. We use the entire dataset with the coarse labels, {x, z}, to train the coarse class network G(x), followed by training multiple fine class network H(x) using only a portion of the dataset.
The inference procedure can be described as follows. Suppose the number of coarse classes in a dataset is C, and each coarse class contains j number of fine classes. Similar to the definition of F (x), we use G(x) to denote the output of the CNN: where j is an positive integer indicating the number of fine classes in the coarse class i. In this setting, the output of the combined neural network is: (3)

Related Work on Adversarial Defence Methods
A large body of defence mechanisms have been proposed for the adversarial setting. Adversarial training (Szegedy et al., 2014) is one of the standard approaches for improving the robustness of deep neural networks against adversarial examples. It is a data augmentation method that replaces unperturbed training data with adversarial examples and updates the network with the replaced data points. Intuitively, this procedure encourages the DNN not to make the same mistakes against an adversary. By adding sufficiently enough adversarial examples, the network gradually becomes robust to the attack it was trained on. Existing adversarial training methods (Szegedy et al., 2014;Goodfellow et al., 2015;Kurakin et al., 2017;Madry et al., 2018;Wong et al., 2019a) differ in the adversaries used in the training. Another related line of adversarial defence methods focuses on regularizing the loss function instead of data augmentation. TRADES (Zhang et al., 2019) introduces a regularization term that penalizes the difference between the output of the model on a training data and its corresponding adversarial example. The regularized loss consists of a standard crossentropy loss on the unperturbed data and a KL-divergence term measuring the difference between the distribution of clean training data and adversarially perturbed training data.

Experiments
In this section, we evaluate the hierarchical adversarial robustness of the HAR network design, incorporating two popular adversarial defence methods: adversarial training with PGD10 adversaries (Madry et al., 2018) and TRADES (Zhang et al., 2019). In this section, we focus on evaluations based on ∞ norm attacks on the CIFAR-100 dataset, and defer evaluations on 2 of CIFAR-100 in Appendix D. Additionally, we include evaluations on the small-sized CIFAR-10 dataset and the medium-sized CIFAR-100-5x5 dataset (

Evaluation Setup
We use network architectures from the ResNet family (He et al., 2016) on the CIFAR-100 dataset. The hierarchical structure of classes within the dataset is illustrated in Table 7. To establish a baseline, we train ResNet50 networks using the following methods: (1)  To further justify the use of separate networks for the coarse and fine classification tasks in HAR, we compare with two natural baselines. ADV-T is a targeted-variant of the PGD adversarial training in Madry et al. (2018). Specifically, given a pair of input from the training set (x, y) and y ∈ z * , the perturbation is computed based on a targeted 10-step PGD attack where the target label is uniformly random sampled from {y | y ∈ z * }. An alternative to HAR is to train a single flat network with a modified hierarchical loss. The idea of incorporating hierarchical structure in losses has been explored in the literature (Cesa-Bianchi et al., 2006;Redmon and Farhadi, 2017;Ge, 2018;Wehrmann et al., 2018), but not yet studied under the robust learning setting. As such, we performed ADV-hCE, i.e., adversarial training using a hierarchical cross-entropy loss: hCE (x, y, z) (F (x), y) + (G(x), z) where F (x) is the prediction of the network of the fine classes. G(x) is the predictions of the coarse classes and its entries are computed by summing the corresponding fine class predictions in F (x).
To differentiate HAR networks from the flat models, we refer the flat models as vanilla models. We use a single ResNet50 for the vanilla model. For the proposed HAR network, we use multiple ResNet10 for the coarse network and the fine networks. We use individual models with a lower capacity for HAR to reduce the difference in the order of magnitude of parameters between a single ResNet50 in the vanilla model and multiple ResNet10 in HAR. Notice that in the experiment with CIFAR-10 and CIFAR-100-5x5, the number of trainable parameters of the vanilla model is larger than that of the HAR network using the selected ResNet architectures. While this is difficult to achieve with CIFAR-100, the goal here is to eliminate the concern of which the improved hierarchical adversarial robustness is obtained solely due to the increasing network complexity. A comparison between the number of trainable parameters is included in Appendix A. Note that in the HAR network, all component networks (CNN and FNNs) are trained using the same adversarial defence approach. As a concrete example, a HAR network trained with TRADES on CIFAR-100 consists of one coarse classifier and twenty fine classifiers where they are all trained using TRADES. For all five methods (Standard, ADV, ADV-T, ADV-hCE and TRADES), networks are trained for a total of 200 epochs, with an initial learning rate of 0.1. The learning rate decays by an order of magnitude at epoch 100 and 150. We used a minibatch size of 128 for testing and training. We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4. For TRADES, we performed a hyperparameter sweep on the strength of the regularization term β and selected one that resulted in the highest accuracy against untargeted ∞ bounded PGD20 attacks. The optimization procedure is used for both the vanilla models and all component models in the HAR network.

Hierarchical Robustness under Untargeted and Targeted Attacks
There are several threat models to consider while evaluating adversarial robustness, regardless of standard or hierarchical robustness. The white-box threat model specifies that the model architecture and network parameters are fully transparent to the attacker (Goodfellow et al., 2015). Despite many white-box attack methods exist, perturbations generated using iterations of PGD remain as one of the most common benchmarks for evaluating adversarial robustness under the white-box setting. Specifically, we perform 20, 50, 100 and 200 iterations of PGD on the entire test set data for the untargeted attacks in Table 2. To evaluate the hierarchical robustness of the model, we perform the worst-case hierarchical adversarial perturbations introduced in Section 2.2. Due to the large number of fine labels in CIFAR-100, the hierarchical attack was performed on 1000 randomly selected test set data. All ∞ -PGD adversarial examples used for all evaluations are generated a step size of /4 (pixel values are normalized to [0, 1]). Along with the two attacks, we also include results on unperturbed testset data (Clean). For clean and untargeted attacks, we report the percentage of correct fine class prediction as fine accuracy, and the percentage of fine class prediction belonging to the correct coarse class as coarse accuracy. For the proposed targeted attack reported in Table 3, the accuracy refers to the percentage of the test set data where the targeted attack fails to alter the final prediction to the desired target, even after iterating through all eligible target labels. It is important to realize that a successful targeted attack implies misclassification for both coarse and fine classes.

Discussions
Before making the comparison between the HAR model and the vanilla model, we make an interesting observation in Table 2: there is a consistent gap between the fine and coarse accuracy for both clean and perturbed inputs. This shows that under the untargeted attack, some of the misclassified inputs at the fine level are actually still classified correctly at the coarse level. In particular, vanilla networks trained with unperturbed training data have close to 0% fine accuracy under untargeted PGD attacks with = 8/255, while nearly 25% of all the misclassified inputs still belong to the correct coarse class. This shows that the untargeted attacks do not provide a good representation of the hierarchical adversarial robustness as an empirical evaluation, and it supports our earlier observation in Section 2.1 which is the main drive for the proposal of a targeted attack specifically designed for reducing coarse accuracy and damaging hierarchical robustness.
Indeed, results in Table 3 show that the proposed worst-case targeted attack severely damages the hierarchical adversarial robustness of all models. On Standard trained models, despite the high hierarchical robustness under untargeted attacks, nearly all of the CIFAR-100 data can be perturbed into a desired target class from another coarse class. Therefore, we emphasize the use of the proposed worst-case targeted attack for a more accurate empirical evaluations for the hierarchical adversarial robustness of the model. For vanilla models trained with ADV and TRADES, we notice that the improved adversarial robustness on fine classes translates to an improvement in hierarchical adversarial robustness.
The result shows that ADV-hCE does not outperform the standard ADV in hierarchical robustness, and thus, we defer its discussions in Appendix F. We discuss ADV-T here, as it serves an excellent case to emphasize that since evaluating robustness empirically using attacks only shows an upper bound on the true robustness of the model, one should focus on the metric that results in the lowest upper bound. For this reason, a network outperforming another on weaker metrics does not imply it has better hierarchical robustness. With this notion, we observed that vanilla models trained using ADV-T shows improved hierarchical robustness against untargeted PGD attacks compared to the original adversarial training method. However, results in Table 3 suggests that the seeming improvement in hierarchical robustness of the ADV-T method does not hold against stronger hierarchical attacks. ADVtrained models outperform ADV-T models on all targeted PGD attacks with = 8/255. Finally, the result shows that ADV-trained HAR network significantly improves the hierarchical robustness against all worst-case targeted attacks compared to the vanilla counterpart. Particularly, we notice a 0.4% increase in accuracy against PGD200 adversaries. For TRADES-robusitified networks, we observe that although the vanilla model outperforms HAR against weaker targeted adversaries (i.e., smaller and small PGD iterations), the gain in hierarchical robustness with HAR rises as the targeted attack becomes stronger. For the strongest PGD200 attack, we notice a 0.1% increase in accuracy compared to the vanilla model.

Hierarchical Robustness under Targeted Attacks based on the Coarse Network
Under the white-box threat model, attackers with a complete knowledge of the internal structure of HAR can also generate perturbations based on the coarse network. During evaluations, we investigate whether the targeted PGD adversaries based on the coarse network are stronger hierarchical adversarial examples compared to the ones generated using the entire network. Such attacks can be understood as finding a more general perturbation Table 4: Accuracy of the HAR network on CIFAR-100 against ∞ bounded targeted attacks ( = 8/255) generated using the coarse network (Coarse). For comparison, the untargeted attack (PGD200 from Table 2) and the attack generated using the entire HAR network (PGD200 from which alters the probability distribution of the coarse class: P (z | x). Similar to the attack proposed in Section 2.2, we perform an iterative, worst-case targeted PGD20 attack based on the coarse neural network. Specifically, we replace (F (x), y) with (G(x), z) in Equation 1, and iterate through all eligible coarse classes as target labels. For example, to generate such attacks for HAR with ADV-trained component networks, the iterative targeted attack is performed based on the ADV-trained coarse network in the original HAR network. Note that there is a distinction between the above attack procedure and a transfer-based attack where the perturbation is transferred from an independently trained source model (Papernot et al., 2017). Since the perturbation is generated using part of the HAR network, such attacks still belongs in the white-box setting. Results in Table 4 show that the perturbations generated using the coarse network are weaker attacks compared to the ones generated using the entire HAR network.

Conclusion
In this work, we introduced a novel concept called hierarchical adversarial examples. For dataset which classes can be further categorized into fine and coarse classes, we defined hierarchical adversarial examples as the ones leading to a misclassification at the coarse level. We proposed a worst-case targeted PGD attack to generate hierarchical adversarial examples. To improve the hierarchical adversarial robustness of DNNs, we proposed the HAR network design, a composite of a coarse network and fine networks where each component network is trained independently by adversarial defence techniques. We empirically showed that HAR leads to a significant increase in hierarchical adversarial robustness on CIFAR-10 and CIFAR-100. The rapid adoption of machine learning applications has also led to an increasing importance in improving the robustness and reliability of such techniques. Mission-critical and safety-critical systems which rely on DNNs in their decision-making process shall incorporate robustness, along with accuracy, in the development process. The introduction of the hierarchical adversarial examples and ways to defend against them is an important step towards a more safe and trustworthy AI system.

A Comparison of trainable model parameters
In our evaluations with CIFAR-100, we use ResNet50 for the vanilla models and multiple ResNet10 for the HAR network. Note that we use architectures with a lower capacity for the individual components in HAR to reduce the difference in the order of magnitude of parameters between a single ResNet50 model and multiple ResNet10 models. For the evaluation with CIFAR-10, we use ResNet34 for the vanilla models, and use multiple ResNet10 for the HAR network. For the evaluation with CIFAR-100-5x5, we use ResNet50 for the vanilla models, and use multiple ResNet10 for the HAR network. In those two cases, the number of trainable parameters of the vanilla model is larger than that of the HAR network. While this is difficult to achieve with CIFAR-100 given its large number of coarse labels, the goal here is to address the concern of which the improved hierarchical adversarial robustness is obtained only due to the increasing network complexity.

B Hyperparameter sweep for TRADES on ResNet10
An important hyperparameter in training with TRADES (Zhang et al., 2019) is β which balances the trade-off between standard accuracy on the clean data and the robust accuracy on the perturbed one. In Zhang et al. (2019), the sweep of β was only performed on more complex models such as WideResNet. As such, we performed a similar hyperparameter sweep to determine the optimal β in TRADES on ResNet10 using the CIFAR-10 dataset. In Table 6, we compared the standard accuracy and the robust accuracy of models trained using TRADES with various value of β. The FGSM and the untargeted PGD20 perturbations are ∞ bounded with ( = 8 255 ). Based on the result, we used β of value 9 while training with TRADES in our evaluations as it resulted in the highest untargeted PGD20 accuracy. We emphasize the main goal of the paper is to demonstrate the novel concept of hierarchical adversarial robustness including techniques to generate and defend against them, rather than reaching for SOTA performance. Therefore, fine-tuning the optimization parameters including the β for the CIFAR-100-5x5 and CIFAR-100 datasets would not be necessary and we simply used the same β for the two datasets. C Hierarchical structure of classes within the CIFAR-10 dataset To demonstrate the effectiveness of the proposed hierarchical attack in Section 2.2 and the improved hierarchical robustness with HAR, we performed an extensive empirical study on three datasets: the small-sized CIFAR-10 dataset with 2 coarse classes and 10 fine classes, the medium-sized CIFAR-100-5x5 dataset with 5 coarse classes and 25 fine classes, and the large-sized CIFAR-100 dataset with 20 coarse classes and 100 fine classes. The hierarchical structure of the datasets are shown in Table 7. In this section, we empirically study the effectiveness of the 2 bounded hierarchical attack and the improved hierarchical robustness from HAR. The 2 bounded worts-case targeted PGD attack is formulated in a similar fashion as the one illustrated in Algorithm 1 and the only difference is that the perturbation generated at each iteration of PGD is ensured to be within the 2 -ball of the original input. Additionally, due to the large number of fine labels in CIFAR-100, the hierarchical attack was performed on 1000 randomly selected test set data. Similar to the setup with ∞ attacks in Section 4.2, we compare various baselines where the model is robustified against 2 attacks. Note that we were not able to achieve reasonable robustness results with TRADES against 2 attacks. One possible reason is that the hyperparameter sweep was based on the ∞ results, as such, the optimal value for β is unsuitable in the 2 setting. For this reason, we omit TRADES in this section. Similar to Section 4.2, we make two important observations. First, we notice that the lowest coarse accuracy for the untargeted PGD attack ( = 0.5) is 53.59% and 48.91% respectively for the ADV-trained vanilla model and the HAR network. In Table 9, we observe that the proposed worst-case targeted PGD attack severely degrades the coarse accuracy for both models. In particular, for the vanilla model, the proposed attack results in a 10% decrease in coarse accuracy. This matches the result with ∞ attacks in Section 4.2 and it shows that the proposed worst-case targeted PGD attack provides a more accurate empirical representation of the hierarchical adversarial robustness of the model. Next, we observe that the targeted adversarial training does not improve the hierarchical robustness compared to the standard adversarial training, and ADV-trained HAR network improves the hierarchical adversarial robustness. Particularly, we observe a 1.1% improvement in the hierarchical robustness against 2 bounded worst-case targeted attacks with the HAR design in comparison to the vanilla network.  E Results on CIFAR-10 and CIFAR-100-5x5 with ∞ bounded attacks In this section, we examine the ∞ bounded hierarchical attack and the improved hierarchical robustness from HAR on the small-sized CIFAR-10 with 2 coarse classes and 10 fine classes, and the medium-sized CIFAR-100-5x5 with 5 coarse classes and 25 fine classes. In Table  10, we compare the HAR network to vanilla baselines using the accuracy on clean data, untargeted PGD20 adversaries and the targeted PGD hierarchical attack proposed in Section 2.2. We observe that HAR network trained with ADV and TRADES significantly improves the robustness against the worst-case targeted attacks compared to the vanilla counterparts. On CIFAR-10, HAR network achieves a 7.52% and 1.47% improvement when trained with ADV and TRADES respectively. On CIFAR-100-5x5, we observe a drastic improvement in hierarchical robustness using HAR: with more than 10% increase in resistance to the worst-case targeted attacks compared to the vanilla models.

F Hierarchical cross-entropy loss
To further justify the use of separate networks for the robust hierarchical classification tasks in HAR, we compare two additional two baselines. In Section 4.2, we showed that the improvement in hierarchical robustness of the ADV-T trained vanilla models does not hold against stronger hierarchical attacks. Another alternative to using separate networks is to train a single flat network with a modified hierarchical loss. The idea of incorporating hierarchical structure in losses has been explored in the literature (Cesa-Bianchi et al., 2006;Redmon and Farhadi, 2017;Ge, 2018;Wehrmann et al., 2018), but not yet studied under the robust learning setting. As such, we proposed ADV-hCE, i.e., adversarial training using a hierarchical cross-entropy loss. First, we define the hierarchical cross-entropy loss as hCE (x, y, z) (F (x), y) + (G(x), z), where F (x) and G(x) are the predictions of the fine classes and coarse classes respectively. In particular, F (x) is the output of the neural network and entries in G(x) are computed by summing the corresponding fine class predictions in F (x). For example, suppose we have four fine classes and two coarse classes and, given an input, the prediction of the network is F (x) = [0.2, 0.4, 0.3, 0.1]. If the first two entries in F (x) correspond to the same coarse classes, then we have G(x) = [0.6, 0.4]. Intuitively, the second term in the hierarchical crossentropy loss can be understood as a penalty for assigning probabilities of fine labels outside of the correct coarse class. As such, we trained two flat ResNet10 models on the CIFAR-10 dataset using adversarial training based on the cross-entropy loss and the hierarchical crossentropy loss respectively. It is important to realize that since adversarial training involves replacing the clean training data with the perturbed data, this means the PGD adversaries used during ADV-hCE are generated based on the hierarchical cross-entropy loss. We observe that although models trained using the modified loss shows improved hierarchical robustness against untargeted PGD attacks generated from both the original and the hierarchical crossentropy loss. However, such an improvement in hierarchical robustness does not hold against stronger targeted hierarchical attacks, captured by the 50% drop in accuracy against the worst-case targeted attack. Table 11: Accuracy of the model trained with the hierarchical cross-entropy loss on CIFAR-10 against ∞ bounded attacks ( = 8/255): the untargeted PGD20 attacks based on the crossentropy loss (Untargeted-CE), the untargeted PGD20 attacks based on the hierarchical crossentropy loss (Untargeted-hCE) and the proposed worst-case hierarchical attack (Targeted).