## Data-driven Verification of DNNs for Object Recognition

Clemens Otte , Yinchong Yang , and Danny Benlin Oswan 1 1 1 2 ,

1- Siemens AG - Technology, 81739 Munich, Germany 2- Technical University of Munich, Informatics, 80333 Munich, Germany

Abstract . The paper proposes a new testing approach for Deep Neural Networks (DNN) using gradient-free optimization to find perturbation chains that successfully falsify the tested DNN, going beyond existing gridbased or combinatorial testing. Applying it to an image segmentation task of detecting railway tracks in images, we demonstrate that the approach can successfully identify weaknesses of the tested DNN regarding particular combinations of common perturbations (e.g., rain, fog, blur, noise) on specific clusters of test images.

## 1 Introduction

Deploying Deep Neural Networks (DNN) in safety-critical fields such as autonomous vehicles [1] requires verifying their robustness and reliability. Formal methods [2, 3] are usually not applicable to the large models used in real-world applications. Thus, verifying the DNN's robustness is mostly done with systematic empirical testing. For example, [4] uses combinatorial (pairwise) testing, which, however, leaves many configurations untested.

Adversarial robustness [3] considers maliciously contrived imperceptible input perturbations, which are often created by following the gradient of the model error on the input pixels. Usually, the likelihood of encountering such adversarial inputs in reality is low and a more practical concern is the robustness towards common (natural) perturbations [5], e.g., blur, fog, snow, rain, noise, and geometric distortions. The question we address in the following is whether these common perturbations can be combined in an adversarial manner that falsifies the tested DNN. Since natural perturbations are in general not differentiable, gradient-based optimization is not applicable. Instead, we propose a new testing approach utilizing gradient-free optimization to guide the perturbation parameters to regions where the model output shows larger model errors, thereby increasing the likelihood of generating test cases that falsify the model. The model output is used as feedback to the optimizer, bypassing the need for gradient information or the DNN's architecture, making the approach scalable for use on large contemporary DNNs.

The approach is evaluated on an image segmentation task in the railway domain. Specifically, we are interested in detecting railway tracks in images taken from the ego-perspective of a train as shown in Fig.1, where a tested Unet model achieves a high Intersection-over-Union (IoU) accuracy on the original images but fails on counterexamples found with the approach described in Sect.3.

Fig. 1: Falsifying a given U-net model, which was trained to segment railway tracks. (a, e) show original (unperturbed) images, (b, f) being the respective model output which is the predicted position of railway tracks. Predictions with probability ≤ 0 5 . are shown in red and do not contribute to the IoU (see eq.2). The approach in Sect.3 created the perturbed images (c, g) where the model output (d, h) shows maximal deterioration.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00783v1-figure-906fea27-369f-4ad1-bd2e-d35ff0cfd780.jpg)

## 2 Related work

Gradient-based adversarial attacks and ℓ p -robustness [6] made the observation that the performance of neural networks is prone to small changes in the feature space that are even unperceivable to human eyes. [7] designed a simple fast gradient sign method (FGSM) attack, making use of only the gradient sign, to fool neural networks efficiently. [8] and [9] proposed iterative approaches known as projected gradient descent (PGD) and Carlini-Wagner (CW) attacks, respectively, to maximize the model's error while keeping the degree of distortion as small as possible. These attacks have been instantly applied to train neural networks ([10, 11]) to resist potential attacks, by performing attacks during each training epoch. This class of attacks operates exclusively in the original feature space, e.g., on pixel values in case of image data, which can be formulated as an ℓ p space. To this end, they are also referred to as ℓ p attack and the model's capability to resist such attack is known as ℓ p -robustness.

Robustness against natural perturbations Instead of the raw features that are perturbed by the adversarial attacks, a second class of data distortion focus on altering the meta features , which is high-level information of the data. For instance, the brightness and contrast may be meta features of an image. The amplitude of all frequencies may be meta features of time series data. Since these perturbations are realistic in that they can be indeed observed in nature, they are also known as natural perturbations. [5] summarized a set of benchmarking perturbation functions, which in turn serves as additional data augmentation methods [12]. Due to the fact that such natural perturbations form an open set and that there is already a very large variety thereof, it is very expensive to directly use them as data augmentation. [13] and [14] developed efficient methods to combine different perturbations into one augmentation to speed up the augmented training.

DeepTest [15] is close to our approach in the sense that natural perturbations are combined. However, it uses a random search over pairs of only two perturbations to maximize the neuron coverage of the tested DNN. Neuron coverage has been criticized as a suitable measure [16]. Instead, we directly maximize the model error using gradient-free optimization on chains of up to six perturbations.

## 3 Gradient-free optimization of perturbations

̸

The objective is to find a sequence of perturbations and their parameters (e.g., mean and standard deviation for Gaussian noise) that maximizes the model error on a set of images. We assume that a set P of perturbations is given ( | P | = 12 in this paper). The optimizer shall draw a subset K K ( | | = 6) of them in a specific order without replacement, and optimize their parameters. The selection of K with the respective order and all individual parameters are encoded in a total parameter vector θ , which is optimized by a gradient-free optimizer as shown in Fig.2, where x denotes an input image and y the corresponding label. For the considered task of image segmentation, y is a binary image mask, indicating whether a pixel belongs to the railway tracks. A chain of perturbations π x, y θ ( | ) yields a perturbed image x θ and its corresponding label y θ with y θ = y only for geometric perturbations (e.g., affine transformations). The model predictions on the original image and the perturbed one are ˆ and ˆ , respectively. y y θ The error e maximized by the optimizer is the average deterioration of the IoU,

$$e = N ^ { - 1 } \sum _ { ( x, y ) } I o U ( \hat { y }, y ) - I o U ( \hat { y } _ { \theta }, y )$$

with N as the number of considered images. The IoU is calculated and averaged over three threshold values, 0.5, 0.9, and 0.99,

$$I o U ( \cdot, y ) = \sum _ { \tau \in \{ 0. 5, 0. 9, 0. 9 9 \} } \frac { | I ( \cdot > \tau ) \wedge y | } { | I ( \cdot > \tau ) \vee y | } / 3,$$

where I denotes a boolean indicator function, to alleviate potential issues with miscalibrated probabilities of the model predictions.

Fig. 2: Optimizing parameter vector θ to maximize model error e

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00783v1-figure-b2dfea01-87aa-4f68-a2d4-54c93b2e05f8.jpg)

## 3.1 Bounding the perturbation strength

To avoid trivial counterexamples, like simply turning the contrast to zero, the strength of the perturbations must be bounded. The bounds are defined upfront by a grid-search performed separately for each perturbation parameter: for each parameter value on a grid, the perturbation is applied to the whole dataset, and the IoU of the model on this perturbed dataset is compared to the IoU on the original, unperturbed dataset. The value that yields an average IoU deterioration close to a 1% is chosen as the maximum limit of that perturbation parameter. That is, we allow every individual perturbation parameter to deteriorate the IoU on average across all images maximally by about 1%.

## 3.2 Clustering the input images

Basically, the approach can be applied to a single image, but the found perturbations are expected to be more meaningful if they are based on multiple images. The optimization results depend on the kind of images. For example, low-light night images are more susceptible to perturbations reducing the contrast. Thus, we run the optimization on clusters of 'similar' images, where similarity is defined by firstly extracting features from each image using a pretrained neural network like EfficientNetB4 [17]. This yields a feature vector per image (with length 1700 for EfficientNetB4), whose dimension is further reduced to 10 by UMAP [18] with a cosine metric before passing it to a common clustering method like kmeans, hierarchical clustering, or DBSCAN. We prefer kmeans as hierarchical clustering created rather imbalanced clusters and the density-based DBSCAN dropped many images as outliers in low-density regions.

## 4 Results

The approach has been tested on the public RailSem19 [19] dataset containing 8500 RGB-pictures with dimension 1920 x 1080 taken from the ego-perspective of a train in different environments. The system under test is a Pytorch U-net image segmentation model, which had been trained on that dataset.

While there are public perturbation tools available (e.g., Albumentations, Imgaug), we use an internal tool, which directly supports the run-length encoded label format in RailSem19.

There are no particular restrictions on the type of perturbations; we just exclude those with too strong effects like histogram equalization, which would make it too easy for the optimizer. Most perturbations have a parameter vector controlling their behavior, e.g., a 'rain' perturbation has a 6-dimensional vector with (opaqueness, size, density, blur, angle, speed) of floats and integers. Perturbations with strong random effects not controlled by a parameter should be avoided as this can hinder the systematic exploration of the parameter space. E.g., 'zoom' originally selected a rectangle position purely randomly; we changed this to include the coordinates in its parameter vector to make them accessible to the optimizer.

Images are assigned to 30 clusters by kmeans as described in Sect.3.2. The gradient-free optimizer uses the two-points differential evolution in [20] with a budget of 5000 iterations on each cluster. With this budget, the verification of a cluster of, e.g., 300 images takes about 20 hours on a GPU processing ca. 20 images per second.

The mean IoU-deterioration per cluster in Fig.3a shows that the model is less robust on dark images, images with wipers (e.g., Fig.1a), and complex urban environments. Note that on every cluster a separate optimization is performed, yielding a cluster-specific chain of perturbations. E.g., in cluster 9, where Fig.1a belongs to, the chain was found to be: ('blur', 'noise', 'affine', 'zoom', 'rain', 'padding') . Fig.3b shows that 'blur' is chosen in every cluster and as one of the first perturbations in the chain. In contrast, 'snow' is added lately (probably to avoid diminishing its effect by later perturbations). 'Fog' only occurs in three clusters but as one of the first perturbations. More details can be found in [21].

(a) Clusters

|   ID |   Size | Description               |   Deter. |
|------|--------|---------------------------|----------|
|    0 |    631 | City streets              |     0.42 |
|    1 |    431 | Rails on gravel           |     0.1  |
|    2 |    249 | Uncertain                 |     0.1  |
|    3 |    313 | Suburbs                   |     0.14 |
|    4 |    386 | Beams, cloudy *           |     0.33 |
|    5 |    383 | Bridges                   |     0.2  |
|    6 |    419 | Snow                      |     0.22 |
|    7 |    446 | Beams, clear skies        |     0.15 |
|    8 |    130 | Trees                     |     0.12 |
|    9 |     97 | Snow with wiper           |     0.44 |
|   10 |    309 | Night / tunnels *         |     0.65 |
|   11 |    388 | Yellow fields             |     0.08 |
|   12 |     92 | Brown soil w/ wiper *     |     0.46 |
|   13 |    145 | Train / railcar           |     0.15 |
|   14 |    209 | Trees                     |     0.14 |
|   15 |    507 | Uncertain                 |     0.16 |
|   16 |    397 | Grasses                   |     0.11 |
|   17 |    104 | Grass with wiper          |     0.08 |
|   18 |    210 | Window w/ wiper *         |     0.42 |
|   19 |    347 | Urban / cars              |     0.33 |
|   20 |    444 | Rails on gravel w/ trees  |     0.08 |
|   21 |    140 | Grass or snows w/ wiper   |     0.26 |
|   22 |    195 | Cloudy evenings *         |     0.34 |
|   23 |    257 | Train / railcar present   |     0.17 |
|   24 |     83 | Suburbs w/ wiper          |     0.16 |
|   25 |    443 | Suburbs                   |     0.14 |
|   26 |    128 | Traffic lights, cloudy    |     0.22 |
|   27 |    118 | Arched bridge             |     0.17 |
|   28 |    190 | Mountains                 |     0.18 |
|   29 |    309 | Square bridge / buildings |     0.1  |

## 5 Conclusion

Gradient-free optimization can find perturbation chains that successfully falsify a given model, going beyond existing grid-based or combinatorial testing. The approach can support the verification by identifying weaknesses of the model regarding specific combinations of perturbations on specific types of images. The

Fig. 3: a) Mean IoU-deterioration per cluster (descriptions with * indicate brightness perturbation being disabled to avoid trivial counterexamples) . Examples (a) and (e) in Fig.1 belong to clusters 9 and 10, respectively. b) Usage of perturbations across clusters.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00783v1-figure-c42908de-5b79-4a9b-b97c-dc0e834c1b86.jpg)

insights can also be used to refine the training process of the model, e.g., by adding perturbed training examples from the clusters with high IoU deterioration.

Acknowledgement We thank Benjamin Hartmann from Siemens Mobility for helpful discussions and for providing the U-net model tested in this approach. This research has received funding from the German Federal Ministry for Economic Affairs and Climate Action under grant agreement 19I21039A (safe.trAIn).

## References

- [1] Tim Fingscheidt, Hanno Gottschalk, and Sebastian Houben, editors. Deep Neural Networks and Data for Automated Driving . Springer, 2022.
- [2] Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J. Kochenderfer. Algorithms for Verifying Deep Neural Networks. Foundations and Trends in Optimization , 2021.
- [3] Mark Huasong Meng, Guangdong Bai, Sin Gee Teo, Zhe Hou, Yan Xiao, Yun Lin, and Jin Song Dong. Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspective. IEEE Trans. Dependable and Secure Computing , 2024.
- [4] Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker, and D. Richard Kuhn. A combinatorial approach to testing deep neural network-based autonomous driving systems. In IEEE Int. Conf. Software Testing, Verification and Validation Workshops , 2021.
- [5] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR , 2019.
- [6] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR , 2014.
- [7] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR , 2015.
- [8] Aleksander Mkadry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. stat , 2017.
- [9] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE symposium on security and privacy , 2017.
- [10] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! NeurIPS , 2019.
- [11] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. ICLR , 2020.
- [12] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. NeurIPS , 2021.
- [13] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR , 2020.
- [14] Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, and Dimitris N. Metaxas. Selfnorm and crossnorm for out-of-distribution robustness. CoRR , abs/2102.02811, 2021.
- [15] Y. Tian, K. Pei, S. Jana, and B. Ray. Deeptest: Automated testing of deep-neuralnetwork-driven autonomous cars. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) , pages 303-314, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.
- [16] Fabrice Harel-Canada, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim. Is neuron coverage a meaningful measure for testing deep neural networks? In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ESEC/FSE 2020, New York, NY, USA, 2020. Association for Computing Machinery.
- [17] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML , 2019.
- [18] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software , 2018.
- [19] Oliver Zendel, Markus Murschitz, Marcel Zeilinger, Daniel Steininger, Sara Abbasi, and Csaba Beleznai. RailSem19: A Dataset for Semantic Rail Scene Understanding. In CVPRW , Long Beach, CA, USA, 2019.
- [20] J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. https: //GitHub.com/FacebookResearch/Nevergrad , 2018.
- [21] Danny Benlin Oswan. Data-driven verification of DNNs for object recognition in the railway domain. Master's thesis, Technical University of Munich, 2023.
Clemens Otte Yinchong Yang Danny Benlin Oswan
Data-driven Verification of DNNs for Object Recognition
The paper proposes a new testing approach for Deep Neural Networks (DNN)
using gradient-free optimization to find perturbation chains that successfully
falsify the tested DNN, going beyond existing grid-based or combinatorial
testing. Applying it to an image segmentation task of detecting railway tracks
in images, we demonstrate that the approach can successfully identify
weaknesses of the tested DNN regarding particular combinations of common
perturbations (e.g., rain, fog, blur, noise) on specific clusters of test
images.