summaries: The work proposes a new attention-based architecture for computer vision (classification). The proposed architecture replaces the self-attention layer in ViT with computational blocks involving the Fourier transform. Due to the efficiency of FFT and IFFT the quadratic complexity of self-attention can be reduced to a log-linear computational cost. In order to boost performance distillation of taken features from a pre-trained vision transformer is used. Results are presented on ImageNet as well as on some transfer learning experiments.

summaries: This paper replaces the self-attention layer with a global filter layer:  a sequence of a 2D FFT, einsum, and then 2D inverse FFT. A benefit of the approach is that the attention computation is reduced from O(n^2) to O(nlogn) time-complexity. This demonstrates that GFNet is Pareto optimal versus other Transformer-like models on ImageNet on an accuracy-FLOPs basis.

summaries: This paper proposes to replace self-attention in vision transformer by a *Global Filter Layer* consisting of a 2D fast Fourier transform (FFT), a point-wise multiplication with learned weights and an 2D inverse FFT. The proposed layer compares favourably with (a) self-attention as it replaces the quadratic dependency on compute cost and memory footprint with a log-linear rate and (b) variants of MLP-mixer as the number of weights scales linearly with the number of pixels instead of quadratically. The performances of the proposed model are convincing both for supervised classification on ImageNet and fine-tuning to downstream task on smaller datasets.

summaries: This paper proposes Global Filter Networks (GFNet), which make use of discrete Fourier transform (DFT) and a global filter layer to mix the input tokens. According to the convolution theorem, the combination of the two operations can be regarded as a depthwise global circular convolution, and thus this work can be viewed as a kind of convolutional network with global convolutional layers. By taking advantage of FFT, the proposed method is faster than previous methods such as self-attention and spatial MLP with comparable performance on ImageNet.

abstractText: Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet.

title: Digital image processing: principles and applications

authors:
Lucas Beyer
Olivier J Hénaff
Alexander Kolesnikov
Xiaohua Zhai
Aäron van den Oord

title: Are we done with imagenet

venue: arXiv preprint arXiv:2006.07159,

authors:
Nicolas Carion
Francisco Massa
Gabriel Synnaeve
Nicolas Usunier
Alexander Kirillov
Sergey Zagoruyko

title: End-to-end object detection with transformers

venue: In ECCV,

authors:
Bowen Cheng
Alexander G. Schwing
Alexander Kirillov

title: Per-pixel classification is not all you need for semantic segmentation

venue: NeurIPS,

authors:
James W Cooley
John W Tukey

title: An algorithm for the machine calculation of complex fourier series

venue: Mathematics of computation,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Caiwen Ding
Siyu Liao
Yanzhi Wang
Zhe Li
Ning Liu
Youwei Zhuo
Chao Wang
Xuehai Qian
Yu Bai
Geng Yuan

title: Circnn: accelerating and compressing deep neural networks using block-circulant weight matrices

venue: In MICRO,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly
Jakob Uszkoreit
Neil Houlsby

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: arXiv preprint arXiv:2010.11929,

authors:
Chengyue Gong
Dilin Wang
Meng Li
Vikas Chandra
Qiang Liu

title: Improve vision transformers training by suppressing over-smoothing

venue: arXiv preprint arXiv:2104.12753,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: arXiv preprint arXiv:1412.6572,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Dan Hendrycks
Thomas Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

venue: arXiv preprint arXiv:1903.12261,

authors:
Dan Hendrycks
Kevin Zhao
Steven Basart
Jacob Steinhardt
Dawn Song

title: Natural adversarial examples

authors:
Elad Hoffer
Tal Ben-Nun
Itay Hubara
Niv Giladi
Torsten Hoefler
Daniel Soudry

title: Augment your batch: Improving generalization through instance repetition

authors:
Zihang Jiang
Qibin Hou
Li Yuan
Daquan Zhou
Xiaojie Jin
Anran Wang
Jiashi Feng

title: Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet

venue: arXiv preprint arXiv:2104.10858,

authors:
Jonathan Krause
Michael Stark
Jia Deng
Li Fei-Fei

title: 3d object representations for fine-grained categorization

venue: In ICCVW,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

authors:
Jae-Han Lee
Minhyeok Heo
Kyung-Rae Kim
Chang-Su Kim

title: Single-image depth estimation based on fourier domain analysis

authors:
James Lee-Thorp
Joshua Ainslie
Ilya Eckstein
Santiago Ontanon

title: Fnet: Mixing tokens with fourier transforms

venue: arXiv preprint arXiv:2105.03824,

authors:
Shaohua Li
Kaiping Xue
Bin Zhu
Chenkai Ding
Xindi Gao
David Wei
Tao Wan

title: Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions

authors:
Zongyi Li
Nikola Kovachki
Kamyar Azizzadenesheli
Burigede Liu
Kaushik Bhattacharya
Andrew Stuart
Anima Anandkumar

title: Fourier neural operator for parametric partial differential equations

authors:
Hanxiao Liu
Zihang Dai
David R So
Quoc V Le

title: Pay attention to mlps

venue: arXiv preprint arXiv:2105.08050,

authors:
Ze Liu
Yutong Lin
Yue Cao
Han Hu
Yixuan Wei
Zheng Zhang
Stephen Lin
Baining Guo

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: arXiv preprint arXiv:2103.14030,

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

venue: arXiv preprint arXiv:1706.06083,

authors:
Xiaofeng Mao
Gege Qi
Yuefeng Chen
Xiaodan Li
Shaokai Ye
Yuan He
Hui Xue

title: Rethinking the design principles of robust vision transformer

venue: arXiv preprint arXiv:2105.07926,

authors:
Maria-Elena Nilsback
Andrew Zisserman

title: Automated flower classification over a large number of classes

venue: In ICVGIP,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An imperative style, high-performance deep learning library

title: Digital image processing algorithms and applications

authors:
Boris T Polyak
Anatoli B Juditsky

title: Acceleration of stochastic approximation by averaging

venue: SIAM journal on control and optimization,

authors:
Ilija Radosavovic
Raj Prateek Kosaraju
Ross Girshick
Kaiming He
Piotr Dollár

title: Designing network design spaces

venue: In CVPR, 2020

authors:
Yongming Rao
Wenliang Zhao
Benlin Liu
Jiwen Lu
Jie Zhou
Cho-Jui Hsieh

title: Dynamicvit: Efficient vision transformers with dynamic token sparsification

venue: In NeurIPS,

authors:
Benjamin Recht
Rebecca Roelofs
Ludwig Schmidt
Vaishaal Shankar

title: Do imagenet classifiers generalize to imagenet? In ICML, pages 5389–5400

authors:
Mingxing Tan
Quoc Le

title: Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105–6114

venue: PMLR,

authors:
Ilya Tolstikhin
Neil Houlsby
Alexander Kolesnikov
Lucas Beyer
Xiaohua Zhai
Thomas Unterthiner
Jessica Yung
Daniel Keysers
Jakob Uszkoreit
Mario Lucic

title: Mlp-mixer: An all-mlp architecture for vision

venue: arXiv preprint arXiv:2105.01601,

authors:
Hugo Touvron
Piotr Bojanowski
Mathilde Caron
Matthieu Cord
Alaaeldin El-Nouby
Edouard Grave
Armand Joulin
Gabriel Synnaeve
Jakob Verbeek
Hervé Jégou

title: Resmlp: Feedforward networks for image classification with data-efficient training

venue: arXiv preprint arXiv:2105.03404,

authors:
Hugo Touvron
Matthieu Cord
Matthijs Douze
Francisco Massa
Alexandre Sablayrolles
Hervé Jégou

title: Training data-efficient image transformers & distillation through attention

venue: arXiv preprint arXiv:2012.12877,

authors:
Hugo Touvron
Matthieu Cord
Alexandre Sablayrolles
Gabriel Synnaeve
Hervé Jégou

title: Going deeper with image transformers

venue: arXiv preprint arXiv:2103.17239,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Wenhai Wang
Enze Xie
Xiang Li
Deng-Ping Fan
Kaitao Song
Ding Liang
Tong Lu
Ping Luo
Ling Shao

title: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021

authors:
Haiping Wu
Bin Xiao
Noel Codella
Mengchen Liu
Xiyang Dai
Lu Yuan
Lei Zhang

title: Cvt: Introducing convolutions to vision transformers

venue: arXiv preprint arXiv:2103.15808,

authors:
Saining Xie
Ross Girshick
Piotr Dollár
Zhuowen Tu
Kaiming He

title: Aggregated residual transformations for deep neural networks

authors:
Yanchao Yang
Stefano Soatto

title: Fda: Fourier domain adaptation for semantic segmentation

authors:
Xumin Yu
Yongming Rao
Ziyi Wang
Zuyan Liu
Jiwen Lu
Jie Zhou

title: Pointr: Diverse point cloud completion with geometry-aware transformers

venue: In ICCV,

authors:
Li Yuan
Yunpeng Chen
Tao Wang
Weihao Yu
Yujun Shi
Zihang Jiang
Francis EH Tay
Jiashi Feng
Shuicheng Yan

title: Tokens-to-token vit: Training vision transformers from scratch on imagenet

venue: arXiv preprint arXiv:2101.11986,

authors:
Sixiao Zheng
Jiachen Lu
Hengshuang Zhao
Xiatian Zhu
Zekun Luo
Yabiao Wang
Yanwei Fu
Jianfeng Feng
Tao Xiang
Philip HS Torr

title: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

venue: arXiv preprint arXiv:2012.15840,

heading: 1 Introduction

text:
The transformer architecture, originally designed for the natural language processing (NLP) tasks [42], has shown promising performance on various vision problems recently [10, 40, 27, 49, 4, 47, 35, 5]. Different from convolutional neural networks (CNNs), vision transformer models use self-attention layers to capture long-term dependencies, which are able to learn more diverse interactions between spatial locations. The pure multi-layer perceptrons (MLP) models [38, 39] further simplify the vision transformers by replacing the self-attention layers with MLPs that are applied across spatial locations. Since fewer inductive biases are introduced, these two kinds of models have the potential to learn more generic and flexible interactions among spatial locations from raw data.
One primary challenge of applying self-attention and pure MLP models to vision tasks is the considerable computational complexity that grows quadratically as the number of tokens increases. Therefore, typical vision transformer style models usually consider a relatively small resolution for the intermediate features (e.g. 14× 14 tokens are extracted from the input images in both ViT [10] and MLP-Mixer [38]). This design may limit the applications of downstream dense prediction tasks like detection and segmentation. A possible solution is to replace the global self-attention with several local self-attention like Swin transformer [27]. Despite the effectiveness in practice, local self-attention brings quite a few hand-made choices (e.g., window size, padding strategy, etc.) and limits the receptive field of each layer.
∗Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this paper, we present a new conceptually simple yet computationally efficient architecture called Global Filter Network (GFNet), which follows the trend of removing inductive biases from vision models while enjoying the log-linear complexity in computation. The basic idea behind our architecture is to learn the interactions among spatial locations in the frequency domain. Different from the self-attention mechanism in vision transformers and the fully connected layers in MLP models, the interactions among tokens are modeled as a set of learnable global filters that are applied to the spectrum of the input features. Since the global filters are able to cover all the frequencies, our model can capture both long-term and short-term interactions. The filters are directly learned from the raw data without introducing human priors. Our architecture is largely based on the vision transformers only with some minimal modifications. We replace the self-attention sub-layer in vision transformers with three key operations: a 2D discrete Fourier transform to convert the input spatial features to the frequency domain, an element-wise multiplication between frequency-domain features and the global filters, and a 2D inverse Fourier transform to map the features back to the spatial domain. Since the Fourier transform is used to mix the information of different tokens, the global filter is much more efficient compared to the self-attention and MLP thanks to the O(L logL) complexity of the fast Fourier transform algorithm (FFT) [7]. Benefiting from this, the proposed global filter layer is less sensitive to the token length L and thus is compatible with larger feature maps and CNN-style hierarchical architectures without modifications. The overall architecture of GFNet is illustrated in Figure 1. We also compare our global filter with prevalent operations in deep vision models in Table 1.
Our experiments on ImageNet verify the effectiveness of GFNet. With a similar architecture, our model outperform the recent vision transformer and MLP models including DeiT [40], ResMLP [39] and gMLP [26]. When using the hierarchical architecture, GFNet can further enlarge the gap. GFNet also works well on downstream transfer learning and semantic segmentation tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness.

heading: 2 Related works

text:
Vision transformers. Since Dosovitskiy et al. [10] introduce transformers to the image classification and achieve a competitive performance compared to CNNs, transformers begin to exhibit their
potential in various vision tasks [3, 4, 49]. Recently, there are a large number of works which aim to improve the transformers [40, 41, 27, 44, 18, 11, 48]. These works either seek for better training strategies [40, 11] or design better architectures [27, 44, 48] or both [41, 11]. However, most of the architecture modification of the transformers [44, 18, 27, 48] introduces additional inductive biases similar to CNNs. In this work, we only focus on the standard transformer architecture [10, 40] and our goal is to replace the heavy self-attention layer (O(L2)) to an more efficient operation which can still model the interactions among different spatial locations without introducing the inductive biases associated with CNNs.
MLP-like models. More recently, there are several works that question the importance of selfattention in the vision transformers and propose to use MLP to replace the self-attention layer in the transformers [38, 39, 26]. The MLP-Mixer [38] employs MLPs to perform token mixing and channel mixing alternatively in each block. ResMLP [39] adopts a similar idea but substitutes the Layer Normalization with an Affine transformation for acceleration. The recently proposed gMLP [26] uses a spatial gating unit to re-weight tokens in the spatial dimension. However, all of the above models include MLPs to mix the tokens spatially, which brings two drawbacks: (1) like the self-attention in the transformers, the spatial MLP still requires computational complexity quadratic to the length of tokens. (2) unlike transformers, MLP models are hard to scale up to higher resolution since the weights of the spatial MLPs have fixed sizes. Our work follows this trend and successfully resolves the above issues in MLP-like models. The proposed GFNet enjoys log-linear complexity and can be easily scaled up to any resolution.
Applications of Fourier transform in vision. Fourier transform has been an important tool in digital image processing for decades [32, 1]. With the breakthroughs of CNNs in vision [14, 13], there are a variety of works that start to incorporate Fourier transform in some deep learning method [24, 46, 9, 22, 6] for vision tasks. Some of these works employ discrete Fourier transform to convert the images to the frequency domain and leverage the frequency information to improve the performance in certain tasks [22, 46], while others utilize the convolution theorem to accelerate the CNNs via fast Fourier transform (FFT) [24, 9]. FFC [6] replaces the convolution in CNNs with an Local Fourier Unit and perform convolutions in the frequency domain. Very recent works also try to leverage Fourier transform to develop deep learning models to solve partial differential equations [25] and NLP tasks [23]. In this work, we propose to use learnable filters to interchange information globally among the tokens in the Fourier domain, inspired by the frequency filters in the digital image processing [32]. We also take advantage of some properties of FFT to reduce the computational costs and the number of parameters.

heading: 3 Method

heading: 3.1 Preliminaries: discrete Fourier transform

text:
We start by introducing the discrete Fourier transform (DFT), which plays an important role in the area of digital signal processing and is a crucial component in our GFNet. For clarity, We first consider the 1D DFT. Given a sequence of N complex numbers x[n], 0 ≤ n ≤ N − 1, the 1D DFT converts the sequence into the frequency domain by:
X[k] = N−1∑ n=0 x[n]e−j(2π/N)kn := N−1∑ n=0 x[n]W knN (3.1)
Algorithm 1 Pseudocode of Global Filter Layer.
# x: the token features, B x H x W x D (where N = H * W) # K: the frequency-domain filter, H x W_hat x D (where W_hat = W // 2 + 1, see Section 3.2 for details)
X = rfft2(x, dim=(1, 2)) X_tilde = X * K x = irfft2(X_tilde, dim=(1, 2))
rfft2/irfft2: 2D FFT/IFFT for real signal
where j is the imaginary unit and WN = e−j(2π/N). The formulation of DFT in Equation (3.1) can be derived from the Fourier transform for continuous signal by sampling in both the time domain and the frequency domain (see Appendix A for details). Since X[k] repeats on intervals of length N , it is suffice to take the value of X[k] at N consecutive points k = 0, 1, . . . , N − 1. Specifically, X[k] represents to the spectrum of the sequence x[n] at the frequency ωk = 2πk/N .
It is also worth noting that DFT is a one-to-one transformation. Given the DFT X[k], we can recover the original signal x[n] by the inverse DFT (IDFT):
x[n] = 1
N N−1∑ k=0 X[k]ej(2π/N)kn. (3.2)
For real input x[n], it can be proved that (see Appendix A) its DFT is conjugate symmetric, i.e., X[N − k] = X∗[k]. The reverse is true as well: if we perform IDFT to X[k] which is conjugate symmetric, a real discrete signal can be recovered. This property implies that the half of the DFT {X[k] : 0 ≤ k ≤ dN/2e} contains the full information about the frequency characteristics of x[n]. DFT is widely used in modern signal processing algorithms for mainly two reasons: (1) the input and output of DFT are both discrete thus can be easily processed by computers; (2) there exist efficient algorithms for computing the DFT. The fast Fourier transform (FFT) algorithms take advantage of the symmetry and periodicity properties of W knN and reduce the complexity to compute DFT from O(N2) to O(N logN). The inverse DFT (3.2), which has a similar form to the DFT, can also be computed efficiently using the inverse fast Fourier transform (IFFT).
The DFT described above can be extend to 2D signals. Given the 2D signal X[m,n], 0 ≤ m ≤ M − 1, 0 ≤ n ≤ N − 1, the 2D DFT of x[m,n] is given by:
X[u, v] = M−1∑ m=0 N−1∑ n=0 x[m,n]e−j2π( um M + vn N ). (3.3)
The 2D DFT can be viewed as performing 1D DFT on the two dimensions alternatively. Similar to 1D DFT, 2D DFT of real input x[m,n] satisfied the conjugate symmetry property X[M − u,N − v] = X∗[u, v]. The FFT algorithms can also be applied to 2D DFT to improve computational efficiency.

heading: 3.2 Global Filter Networks

text:
Overall architecture. Recent advances in vision transformers [10, 40] demonstrate that models based on self-attention can achieve competitive performance even without the inductive biases associated with the convolutions. Henceforth, there are several works [39, 38] that exploit approaches (e.g., MLPs) other than self-attention to mix the information among the tokens. The proposed Global Filter Networks (GFNet) follows this line of work and aims to replace the heavy self-attention layer (O(N2)) with a simpler and more efficient one. The overall architecture of our model is depicted in Figure 1. Our model takes as an input H ×W non-overlapping patches and projects the flattened patches into L = HW tokens with dimension D. The basic building block of GFNet consists of: 1) a global filter layer that can exchange spatial information efficiently (O(L logL)); 2) a feedforward network (FFN) as in [10, 40]. The output tokens of the last block are fed into a global average pooling layer followed by a linear classifier.
Global filter layer. We propose global filter layer as an alternative to the self-attention layer which can mix tokens representing different spatial locations. Given the tokens x ∈ RH×W×D, we first
perform 2D FFT (see Section 3.1) along the spatial dimensions to convert x to the frequency domain:
X = F [x] ∈ CH×W×D, (3.4) where F [·] denotes the 2D FFT. Note that X is a complex tensor and represents the spectrum of x. We can then modulate the spectrum by multiplying a learnable filter K ∈ CH×W×D to the X:
X̃ = K X, (3.5)
where is the element-wise multiplication (also known as the Hadamard product). The filter K is called the global filter since it has the same dimension with X , which can represent an arbitrary filter in the frequency domain. Finally, we adopt the inverse FFT to transform the modulated spectrum X̃ back to the spatial domain and update the tokens:
x← F−1[X̃]. (3.6)
The formulation of the global filter layer is motivated by the frequency filters in the digital image processing [32], where the global filter K can be regarded as a set of learnable frequency filters for different hidden dimensions. It can be proved (see Appendix A) that the global filter layer is equivalent to a depthwise global circular convolution with the filter size H ×W . Therefore, the global filter layer is different from the standard convolutional layer which adopts a relatively small filter size to enforce the inductive biases of the locality. We also find although the proposed global filter can also be interpreted as a spatial domain operation, the filters learned in our networks exhibit more clear patterns in the frequency domain than the spatial domain, which indicates our models tend to capture relation in the frequency domain instead of spatial domain (see Figure 4). Note that the global filter implemented in the frequency domain is also much more efficient compared to the spatial domain, which enjoys a complexity of O(DL logL) while the vanilla depthwise global circular convolution in the spatial domain has O(DL2) complexity. We will also show that the global filter layer is better than its local convolution counterparts in the experiments.
It is also worth noting that in the implementation, we make use of the property of DFT to reduce the redundant computation. Since x is a real tensor, its DFT X is conjugate symmetric, i.e. X[H − u,W − v, :] = X∗[H,W, :]. Therefore, we can take only the half of the values in the X but preserve the full information at the same time:
Xr = X[:, 0 : Ŵ ] := Fr[x], Ŵ = dW/2e, (3.7)
where Fr denotes the 2D FFT for real input. In this way, we can implement the global filter as Kr ∈ CH×Ŵ×D, which can reduce half the parameters. This can also ensure F−1r [Kr Xr] is a real tensor, thus it can be added directly to the input x. The global filter layer can be easily in modern deep learning frameworks (e.g., PyTorch [31]), as is shown in Algorithm 1. The FFT and IFFT are well supported by GPU and CPU thanks to the acceleration libraries like cuFFT and mkl-fft, which makes our models perform well on hardware.
Relationship to other transformer-style models. The GFNet follows the line of research about the exploration of approaches to mix the tokens. Compared to existing architectures like vision transformers and pure MLP models, we exhibit that GFNet has several favorable properties: 1) GFNet is more efficient. The complexity of both the vision transformers [10, 40, 41] and the MLP models [38, 39] is O(L2). Different from them, global filter layer only consists an FFT (O(L logL)), an element-wise multiplication (O(L)) and an IFFT (O(L logL)), which means the total computational complexity is O(L logL). 2) Although pure MLP models are simpler compared to transformers, it is hard to fine-tune them on higher resolution (e.g., from 224 × 224 resolution to 384× 384 resolution) since they can only process a fixed number of tokens. As opposed to pure MLP models, we will show that our GFNet can be easily scaled up to higher resolution. Our model is more flexible since both the FFT and the IFFT have no learnable parameters and can process sequences with arbitrary length. We can simply interpolate the global filter K to K ′ ∈ CH′×W ′×D for different inputs, where H ′ ×W ′ is the target size. The interpolation is reasonable due to the property of DFT. Each element of the global filter K[u, v] corresponds to the spectrum of the filter at ωu = 2πu/H, ωv = 2πv/W and thus, the global filter K can be viewed as a sampling of a continuous spectrum K(ωu, ωv), where ωu, ωv ∈ [0, 2π]. Hence, changing the resolution is equivalent to changing the sampling interval of K(ωu, ωv). Therefore, we only need to perform interpolation to shift from one resolution to another.
We also notice recently a concurrent work FNet [23] leverages Fourier transform to mix tokens. Our work is distinct from FNet in three aspects: (1) FNet performs FFT to the input and directly adds the real part of the spectrum to the input tokens, which blends the information from different domains (spatial/frequency) together. On the other hand, GFNet draws motivation from the frequency filters, which is more reasonable. (2) FNet only keeps the real part of the spectrum. Note that the spectrum of real input is conjugate symmetric, which means the real part is exactly symmetric and thus contains redundant information. Our GFNet, however, utilizes this property to simplify the computation. (3) FNet is designed for NLP tasks, while our GFNet focuses on vision tasks. In our experiments, we also implement the FNet and show that our model outperforms it.
Architecture variants. Due to the limitation from the quadratic complexity in the self-attention, vision transformers [10, 40] are usually designed to process a relatively small feature map (e.g., 14×14). However, our GFNet, which enjoys log-linear complexity, avoids that problem. Since in our GFNet the computational costs do not grow such significantly when the feature map size increases, we can adopt a hierarchical architecture inspired by the success of CNNs [21, 14]. Generally speaking, we can start from a large feature map (e.g., 56× 56) and gradually perform downsampling after a few blocks. In this paper, we mainly investigate two kinds of variants of GFNet: transformer-style models with a fixed number of tokens in each block and CNN-style hierarchical models with gradually downsampled tokens. For transformer-style models, we begin with a 12-layer model (GFNet-XS) with a similar architecture with DeiT-S and ResMLP-12. Then, we obtain 3 variants of the model (GFNet-Ti, GFNet-S and GFNet-B) by simply adjusting the depth and embedding dimension, which have similar computational costs with ResNet-18, 50 and 101 [14]. For hierarchical models, we also design three models (GFNet-H-Ti, GFNet-H-S and GFNet-H-B) that have these three levels of complexity following the design of PVT [43]. We use 4× 4 patch embedding to form the input tokens and use a non-overlapping convolution layer to downsample tokens following [43, 27]. Unlike PVT [43] and Swin [27], we directly apply our building block on different stages without any modifications. The detailed architectures are summarized in Table 2.

heading: 4 Experiments

text: We conduct extensive experiments to verify the effectiveness of our GFNet. We present the main results on ImageNet [8] and compare them with various architectures. We also test our models on the downstream transfer learning datasets including CIFAR-10/100 [20], Stanford Cars [19] and Flowers-102 [30]. Lastly, we investigate the efficiency and robustness of the proposed models and provide visualization to have an intuitive understanding of our method.

heading: 4.1 ImageNet Classification

text:
Setups. We conduct our main experiments on ImageNet [8], which is a widely used large-scale benchmark for image classification. ImageNet contains roughly 1.2M images from 1,000 categories. Following common practice [14, 40], we train our models on the training set of ImageNet and report the single-crop top-1 accuracy on 50,000 validation images. To fairly compare with previous works [40, 39], we follow the most training details for our models and do not add extra regularization methods like [18]. Different from [40], we does not use EMA model [33], RandomEarse [50] and
repeated augmentation [17], which are important to train DeiT while sightly hurting the performance of our models. We set the gradient clipping norm to 1 for all of our models. During finetuning at the higher resolution, we use the hyper-parameters suggested by the implementation of [40] and train the model for 30 epochs. All of our models are trained on a single machine with 8 GPUs. More details can be found in Appendix B.
Comparisons with transformer-style architectures. The results are presented in Table 3. We compare our method with different transformer-style architectures for image classification including
vision transformers (DeiT [40]) and MLP-like models (ResMLP [39] and gMLP [26]) that have similar complexity and number of parameters. We see that our method can clearly outperform recent MLP-like models such as ResMLP [39] and gMLP [26], and show similar performance to DeiT. Specifically, GFNet-XS outperforms ResMLP-12 by 2.0% while having slightly fewer FLOPs. GFNet-S also achieves better top-1 accuracy compared to gMLP-S and DeiT-S. Our tiny model is significantly better compared to both DeiT-Ti (+2.4%) and gMLP-Ti (+2.6%) with the similar level of complexity.
Comparisons with hierarchical architectures. We compare different kinds of hierarchical models in Figure 4. ResNet [14] is the most widely used convolutional model while RegNet [34] is a family of carefully designed CNN models. We also compare with recent hierarchical vision transformers PVT [43] and Swin [27]. Benefiting from the log-linear complexity, GFNet-H models show significantly better performance than ResNet, RegNet and PVT and achieve similar performance with Swin while having a much simpler and more generic design.
Fine-tuning at higher resolution. One prominent problem of MLP-like models is that the feature resolution is not adjustable. On the contrary, the proposed global filter is more flexible. We demonstrate the advantage of GFNet by finetuning the model trained at 224 × 224 resolution to higher resolution following the practice in vision transformers [40]. As shown in Table 3, our model can easily adapt to higher resolution with only 30 epoch finetuning and achieve better performance.

heading: 4.2 Transfer learning

text: To test the generality of our architecture and the learned representation, we evaluate GFNet on a set of commonly used transfer learning benchmark datasets including CIFAR-10 [20], CIFAR-100 [20], Stanford Cars [19] and Flowers-102 [30]. We follow the setting of previous works [37, 10, 40, 39], where the model is initialized by the ImageNet pre-trained weights and finetuned on the new datasets. We evaluate the transfer learning performance of our basic model and best model. The results are presented in Table 5. The proposed models generally work well on downstream datasets. GFNet models outperform ResMLP models by a large margin and achieve very competitive performance with state-of-the-art EfficientNet-B7. Our models also show competitive performance compared to state-of-the-art CNNs and vision transformers.

heading: 4.3 Analysis and visualization

text:
Efficiency of GFNet. We demonstrate the efficiency of our GFNet in Figure 2, where the models are compared in theoretical FLOPs, actual latency and peak memory usage on GPU. We test a single building block of each model (including one token mixing layer and one FFN) with respect to the different numbers of tokens and set the feature dimension and batch size to 384 and 32 respectively. The self-attention model quickly runs out of memory when feature resolution exceeds 562, which is also the feature resolution of our hierarchical model. The advantage of the proposed architecture becomes larger as the resolution increases, which strongly shows the potential of our model in vision tasks requiring high-resolution feature maps.
0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 GFLOPs
72
74
76
78
80
82
Im ag
eN et
T op
-1 A
cc (%
)
GFNet-Ti
GFNet-XS
GFNet-S
GFNet-B
GFNet-S 384
DeiT-Ti
DeiT-S
DeiT-B
ResMLP-12
ResMLP-24
ResMLP-36
gMLP-Ti
gMLP-S
gMLP-B
GFNet DeiT ResMLP gMLP
Figure 3: ImageNet acc. vs model complexity.
Table 6: Comparisons among the GFNet and other variants based on the transformer-like architecture on ImageNet. We show that GFNet outperforms the ResMLP [39], FNet [23] and models with local depthwise convolutions. We also report the number of parameters and theoretical complexity in FLOPs.
Model Acc Param FLOPs(%) (M) (G)
DeiT-S [40] 79.8 22 4.6
Local Conv (3× 3) 77.7 15 2.8 Local Conv (5× 5) 78.1 15 2.9 Local Conv (7× 7) 78.2 15 2.9 ResMLP [39] 76.6 15 3.0 FNet [23] 71.2 15 2.9
GFNet-XS 78.6 16 2.9
Complexity/accuracy trade-offs. We show the computational complexity and accuracy trade-offs of various transformer-style architectures in Figure 3. It is clear that GFNet achieves the best trade-off among all kinds of models.
Ablation study on the global filter. To more clearly show the effectiveness of the proposed global filters, we compare GFNet-XS with several baseline models that are equipped with different token mixing operations. The results are presented in Table 6. All models have a similar building block ( token mixing layer + FFN ) and the same feature dimension of D = 384. We also implement the recent FNet [23] for comparison, where a 1D FFT on feature dimension and a 2D FFT on spatial dimensions are used to mix tokens. As shown in Table 6, our method outperforms all baseline methods except DeiT-S that has 64% higher FLOPs.
Robustness & generalization ability. Inspired by the [29], we further conduct experiments to evaluate the robustness and the generalization ability of the GFNet. For robustness, we consider ImageNet-A, ImageNet-C, FGSM and PGD. ImageNet-A [16] (IN-A) is a challenging dataset that contains natural adversarial examples. ImageNet-C [15] (IN-C) is used to validate the robustness of the model under various types of corruption. We use the mean corruption error (mCE, lower is better) on ImageNet-C as the evaluation metric. FGSM [12] and PGD [28] are two widely used algorithms that are targeted to evaluate the adversarial robustness of the model by single-step attack and multistep attack, respectively. For generalization ability, we adopt two variants of ImageNet validation set: ImageNet-V2 [36] (IN-V2) and ImageNet-Real [2] (IN-Real). ImageNet-V2 is a re-collected version of ImageNet validation set following the same data collection procedure of ImageNet, while ImageNet-Real contains the same images as ImageNet validation set but has reassessed labels. We compare GFNet-S with various baselines in Table 7 including CNNs, Transformers and MLP-like architectures and find the GFNet enjoys both favorable robustness and generalization ability.
Visualization. The core operation in GFNet is the element-wise multiplication between frequencydomain features and the global filter. Therefore, it is easy to visualize and interpret. We visualize the frequency domain filters as well as their corresponding spatial domain filters in Figure 4. The learned global filters have more clear patterns in the frequency domain, where different layers have different characteristics. Interestingly, the filters in the last layer particularly focus on the low-frequency component. The corresponding filters in the spatial domain are less interpretable for humans.

heading: 5 Conclusion

text: We have presented the Global Filter Network (GFNet), which is a conceptually simple yet computationally efficient architecture for image classification. Our model replaces the self-attention sub-layer in vision transformer with 2D FFT/IFFT and a set of learnable global filters in the frequency domain. Benefiting from the token mixing operation with log-linear complexity, our architecture is highly efficient. Our experimental results demonstrated that GFNet can be a very competitive alternative to vision transformers, MLP-like models and CNNs in accuracy/complexity trade-offs.

heading: Acknowledgment

text: This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 62125603, Grant 61822603, Grant U1813218, Grant U1713214, in part by Beijing Academy of Artificial Intelligence (BAAI), and in part by a grant from the Institute for Guo Qiang, Tsinghua University.

title: Global Filter Networks for Image Classification


summaries: This paper tests methods for training a classifier for predicting trustworthiness-- formulated here as predicting whether a given image will be classified correctly or not by an target classifier -- on large scale datasets and finds that using existing loss functions like uncertainty estimates/focal loss on this binary classification task cannot detect untrustworthy samples, all samples are classified as positive resulting in 100% FPR (0% TNR). They attempt to fix this shortcoming by introducing a steep slope loss which essentially separates features w.r.t. correct and incorrect predictions from each other, and the loss is bounded in its magnitude. The paper extensively tests whether its possible to predict trustworthiness with two oracle/target classifier combinations on Imagenet and its stylized/adversarial test sets.

summaries: The paper proposes the steep slope loss for learning trustworthiness under a supervised training setup. A pre-trained neural network is further trained on the steep slope loss to learn trustworthiness as a binary classification problem. The approach is evaluated on ImageNet for a ResNet and a ViT model. In addition, evaluations on MNIST and CIFAR-10 are provided for comparisons with the related work. In all experiments, the proposed approach shows good performance.

summaries: The paper proposes Steep Slope loss, a loss function that is designed to improve performance in the binary classification task of prediction trustworthiness. The paper reports that common loss functions used in this domain such as cross-entropy loss, focal loss, and true classification probability loss (TCP), all suffer from an issue where correct predictions tend to dominate over incorrect predictions, due to issues of class imbalance and evaluations on simple datasets where classifiers have achieved very high accuracy.   Steep Slope loss solves this issue by controlling the slope parameters on both the negative and positive classification thresholds with two hyperparameters that control the derivative and gradient updates for datapoints that are correctly classified and incorrectly classified separately. These two controlled slopes are a function of the signed distance of the output of the classifier from the hyperplane of classification.  The paper evaluates this loss on the MNIST, CIFAR, and Imagenet datasets and shows empirical improvements on multiple binary classification metrics such as AUPRC, AUROC, and FPR95, and importantly, SS loss shows marked improvements in the TNR, therefore being more robust to detecting incorrectly classified points.

summaries: The authors find that past trustworthiness predicting methods are prone to overfitting to the training set where correct predictions are dominant. The label imbalance makes the trustworthiness predictors over-confident to even incorrect predictions. To solve the problem, this work proposes a new training loss named steep slope loss, which aims at improving the generalizability of trustworthiness predictors. The intuition behind this is to emphasize the negative class during training. The experiments are conducted in ImageNet. The results show the shortcomings of past methods with cross-entropy loss, focal loss, and TCP loss and then demonstrate the effectiveness of the proposed method.

abstractText: Understanding the trustworthiness of a prediction yielded by a classifier is critical for the safe and effective use of AI models. Prior efforts have been proven to be reliable on small-scale datasets. In this work, we study the problem of predicting trustworthiness on real-world large-scale datasets, where the task is more challenging due to high-dimensional features, diverse visual concepts, and a large number of samples. In such a setting, we observe that the trustworthiness predictors trained with prior-art loss functions, i.e., the cross entropy loss, focal loss, and true class probability confidence loss, are prone to view both correct predictions and incorrect predictions to be trustworthy. The reasons are two-fold. Firstly, correct predictions are generally dominant over incorrect predictions. Secondly, due to the data complexity, it is challenging to differentiate the incorrect predictions from the correct ones on real-world large-scale datasets. To improve the generalizability of trustworthiness predictors, we propose a novel steep slope loss to separate the features w.r.t. correct predictions from the ones w.r.t. incorrect predictions by two slide-like curves that oppose each other. The proposed loss is evaluated with two representative deep learning models, i.e., Vision Transformer and ResNet, as trustworthiness predictors. We conduct comprehensive experiments and analyses on ImageNet, which show that the proposed loss effectively improves the generalizability of trustworthiness predictors. The code and pre-trained trustworthiness predictors for reproducibility are available at https://github.com/luoyan407/predict_trustworthiness.

authors:
Daniel Leidner
Christoph Borst
Alexander Dietrich
Michael Beetz
Alin Albu-Schäffer

title: Classifying compliant manipulation tasks for automated planning in robotics

venue: In IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
Antoine Caté
Lorenzo Perozzi
Erwan Gloaguen
Martin Blouin

title: Machine learning as a tool for geologists

venue: The Leading Edge,

authors:
José Antonio Sanz
Mikel Galar
Aranzazu Jurio
Antonio Brugos
Miguel Pagola
Humberto Bustince

title: Medical diagnosis of cardiovascular diseases using an interval-valued fuzzy rule-based classification system

venue: Applied Soft Computing,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: ImageNet classification with deep convolutional neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Mingxing Tan
Quoc V. Le

title: EfficientNet: Rethinking model scaling for convolutional neural networks

venue: In International Conference on Machine Learning,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly
Jakob Uszkoreit
Neil Houlsby

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: In International Conference on Learning Representations,

authors:
Heinrich Jiang
Been Kim
Melody Guan
Maya Gupta

title: To trust or not to trust a classifier

authors:
Charles Corbière
Nicolas Thome
Avner Bar-Hen
Matthieu Cord
Patrick Pérez

title: Addressing failure prediction by learning model confidence

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: ImageNet: A large-scale hierarchical image database

authors:
Mitra Basu
Tin Kam Ho

title: Data complexity in pattern recognition

venue: Springer Science & Business Media,

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

authors:
Foster J. Provost
Tom Fawcett
Ron Kohavi

title: The case against accuracy estimation for comparing induction algorithms

venue: In Proceedings of the Fifteenth International Conference on Machine Learning,

authors:
Ian Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

authors:
Anh Nguyen
Jason Yosinski
Jeff Clune

title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

title: Machine learning: a probabilistic perspective

venue: MIT press,

authors:
Tsung-Yi Lin
Priya Goyal
Ross Girshick
Kaiming He
Piotr Dollár

title: Focal loss for dense object detection

venue: In IEEE International Conference on Computer Vision,

title: An overview of statistical learning theory

venue: IEEE Transactions on Neural Networks,

authors:
Mehryar Mohri
Afshin Rostamizadeh
Ameet Talwalkar

title: Foundations of machine learning

title: Regression models and life-tables

venue: Journal of the Royal Statistical Society: Series B (Methodological),

authors:
Weiyang Liu
Yandong Wen
Zhiding Yu
Meng Yang

title: Large-margin softmax loss for convolutional neural networks

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

authors:
Jooyoung Moon
Jihyo Kim
Younghak Shin
Sangheum Hwang

title: Confidence-aware learning for deep neural networks

authors:
Chen Huang
Yining Li
Chen Change Loy
Xiaoou Tang

title: Learning deep representation for imbalanced classification

authors:
Yiru Wang
Weihao Gan
Jie Yang
Wei Wu
Junjie Yan

title: Dynamic curriculum learning for imbalanced data classification

venue: In IEEE/CVF International Conference on Computer Vision,

authors:
Kaidi Cao
Colin Wei
Adrien Gaidon
Nikos Arechiga
Tengyu Ma

title: Learning imbalanced datasets with label-distribution-aware margin loss

authors:
Yin Cui
Menglin Jia
Tsung-Yi Lin
Yang Song
Serge Belongie

title: Class-balanced loss based on effective number of samples

venue: In IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Jaehyung Kim
Jongheon Jeong
Jinwoo Shin

title: M2m: Imbalanced classification via majorto-minor translation

authors:
Koby Crammer
Jaz Kandola
Yoram Singer

title: Online classification on a budget

authors:
Yahav Bechavod
Christopher Jung
Steven Z. Wu

title: Metric-free individual fairness in online learning

authors:
Robert Geirhos
Patricia Rubisch
Claudio Michaelis
Matthias Bethge
Felix A. Wichmann
Wieland Brendel

title: ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

authors:
Cassidy Laidlaw
Soheil Feizi

title: Functional adversarial attacks

authors:
Dan Hendrycks
Thomas Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

authors:
S. Kullback
R.A. Leibler

title: On Information and Sufficiency

venue: The Annals of Mathematical Statistics,

title: On a measure of divergence between two multinomial populations

venue: Sankhyā: The Indian Journal of Statistics,

title: Learning multiple layers of features from tiny images

venue: Master’s thesis, University of Toronto,

authors:
Yonatan Geifman
Ran El-Yaniv

title: Selective classification for deep neural networks

heading: 1 Introduction

text:
Classification is a ubiquitous learning problem that categorizes objects according to input features. It is widely used in a range of applications, such as robotics [1], environment exploration [2], medical diagnosis [3], etc. In spite of the successful development of deep learning methods in recent decades, high-performance classifiers would still have a chance to make mistakes due to the improvability of models and the complexity of real-world data [4, 5, 6, 7].
To assess whether the prediction yielded by a classifier can be trusted or not, there are growing efforts towards learning to predict trustworthiness [8, 9]. These methods are evaluated on small-scale datasets, e.g., MNIST [10], where the data is relatively simple and existing classifiers have achieved high accuracy (> 99%). As a result, there are a dominant proportion of correct predictions and the trustworthiness predictors are prone to classify incorrect predictions as trustworthy predictions. The characteristics that the simple data is easy-to-classify aggravate the situation. To further understand the prowess of predicting trustworthiness, we study this problem on the real-world large-scale datasets, i.e., ImageNet [11]. This is a challenging theme for classification in terms of boundary complexity, class ambiguity, and feature dimensionality [12]. As a result, failed predictions are inevitable.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A general illustration of predicting trustworthiness [9, 13] is shown in Fig. 1a. The trustworthiness predictor [13] that is based on the maximum confidence have been proven to be unreliable [8, 14, 15, 16]. Instead, Corbiere et al. [9] propose the true class probability (TCP) that uses the confidence w.r.t. the ground-truth class to determine whether to trust the classifier’s prediction or not. Nevertheless, the classification confidence is sensitive to the data. As shown in Fig. 1b, TCP predicts that all the incorrect predictions (0.9% in predictions) are trustworthy on MNIST [10] and predicts that all the incorrect predictions (∼16% in predictions) are trustworthy on ImageNet [11]. To comprehensively understand this problem, we follow the learning scheme used in [9] and use two state-of-the-art backbones, i.e., ViT [7] and ResNet [5], as the trustworthiness predictors. For simplicity, we call the “trustworthiness predictor" an oracle. We find that the oracles trained with cross entropy loss [17], focal loss [18], and TCP confidence loss [9] on ImageNet are prone to overfit the training samples, i.e., the true positive rate (TPR) is close to 100% while the true negative rate (TNR) is close to 0%. To improve the generalizability of oracles, we propose a novel loss function named the steep slope loss. The proposed steep slope loss consists of two slide-like curves that cross with each other and face in the opposite direction to separate the features w.r.t. trustworthy and untrustworthy predictions. It is tractable to control the slopes by indicating the heights of slides. In this way, the proposed loss is able to be flexible and effective to push the features w.r.t. correct and incorrect predictions to the well-classified regions.
Predicting trustworthiness is similar to as well as different from conventional classification tasks. On one hand, predicting trustworthiness can be formulated as a binary classification problem. On the other hand, task-specific semantics are different between the classification task and predicting trustworthiness. The classes are referred to visual concepts, such as dog, cat, etc., in the classification task, while the ones in predicting trustworthiness are abstract concepts. The trustworthiness could work on top of the classes in the classification task. In other words, the classes in the classification task are specific and closed-form, while trustworthiness is open-form and is related to the classes in the classification task.
The contribution of this work can be summarized as follows.
• We study the problem of predicting trustworthiness with widely-used classifiers on ImageNet. Specifically, we observe that a major challenge of this learning task is that the cross entropy loss, focal loss, and TCP loss are prone to overfit the training samples, where correct predictions are dominant over incorrect predictions.
• We propose the steep slope loss function that improves the generalizability of trustworthiness predictors. We conduct comprehensive experiments and analyses, such as performance on both small-scale and large-scale datasets, analysis of distributions separability, comparison to the class-balanced loss, etc., which verify the efficacy of the proposed loss.
• To further explore the practicality of the proposed loss, we train the oracle on the ImageNet training set and evaluate it on two variants of ImageNet validation set, i.e., the stylized validation set and the adversarial validation set. The two variants’ domains are quite different
from the domain of the training set. We find that the learned oracle is able to consistently differentiate the trustworthy predictions from the untrustworthy predictions.

heading: 2 Preliminaries

text:
In this section, we first recap how a deep learning model learns in the image classification task. Then, we show how the task of predicting trustworthiness connects to the classification task.
Supervised Learning for Classification. In classification tasks, given a training sample, i.e., image x ∈ Rm and corresponding ground-truth label y ∈ Y = {1, . . . ,K}, we assume that samples are drawn i.i.d. from an underlying distribution. The goal of the learning task is to learn to find a classifier f (cls)(·; θ′) with training samples for classification. θ′ is the set of parameters of the classifier. Let f (cls) θ′ (·) = f (cls)(·; θ′). The optimization problem is defined as
f ∗(cls) θ′ = argmin
f (cls)
θ′
R̂(f (cls)θ′ , ` (cls), Dtr), (1)
where f∗(cls)θ′ is the learned classifier, R̂ is the empirical risk, `(cls) is a loss function for classification, and Dtr is the set of training samples.
Supervised Learning for Predicting Trustworthiness. In contrast to the learning task for classification, which is usually a multi-class single-label classification task [4, 5, 6, 7], learning to predict trustworthiness is a binary classification problem, where the two classes are positive (i.e., trustworthy) or negative (i.e., untrustworthy). Similar to [9], given a pair (x, y) and a classifier f (cls)θ′ , we define the ground-truth label o for predicting trustworthiness as
o = { 1, if argmax f (cls)θ′ (x) = y 0, otherwise
(2)
In other words, the classifier correctly predicts the image’s label so the prediction is trustworthy in hindsight, otherwise the prediction is untrustworthy.
The learning task for predicting trustworthiness follows a similar learning framework in the classification task. Let fθ(·) be an oracle (i.e., a trustworthiness predictor). A generic loss function ` : Rm×R → R≥0, where R≥0 is a non-negative space and m is the number of classes. Given training samples (x, y) ∈ Dtr, the optimization problem for predicting trustworthiness is defined as
f∗θ = argmin fθ
1
|Dtr| |Dtr|∑ i=1 `(fθ(xi), oi), (3)
where |Dtr| is the cardinality of Dtr. Particularly, we consider two widely-used loss functions for classification and the loss function used for training trustworthiness predictors as baselines. They are the cross entropy loss [17], focal loss [18], and TCP confidence loss [9]. Let p(o = 1|θ,x) = 1/(1 + exp(−z)) be the trustworthiness confidence, where z ∈ R is the descriminative feature produced by the oracle, i.e., z = fθ(x). The three loss functions can be written as
`CE(fθ(x), o) = −o · log p(o = 1|θ,x)− (1− o) · log(1− p(o = 1|θ,x)), (4) `Focal(fθ(x), o) = −o · (1− p(o = 1|θ,x))γ log p(o = 1|θ,x)−
(1− o) · (p(o = 1|θ,x))γ log(1− p(o = 1|θ,x)), (5)
`TCP (fθ(x), y) = (fθ(x)− p(ŷ = y|θ′,x))2. (6) In the focal loss, γ is a hyperparameter. In the TCP confidence loss, ŷ is the predicted label and p(ŷ = y|θ′,x) is the classification probability w.r.t. the ground-truth class. Consequently, the learned oracle would yield z to generate the trustworthiness confidence. In the cases of `CE and `Focal, the oracle considers a prediction is trustworthy if the corresponding trustworthiness confidence is greater than the positive threshold 0.5, i.e., p(o = 1|θ,x) > 0.5. The predictions whose trustworthiness confidences are equal to or lower than the negative threshold 0.5 are viewed to be untrustworthy. In the case of `TCP , the positive threshold is also 0.5, but the negative threshold correlates to the number of classes in the classification task. It is defined as 1/K in [9].

heading: 3 Methodology

text: In this section, we first introduce the overall learning framework for predicting trustworthiness. Then, we narrow down to the proposed steep slope loss function. At last, we provide the generalization bound that is related to the proposed steep slope loss function.

heading: 3.1 Overall Design

text:
Corbière et al. [9] provide a good learning scheme for predicting trustworthiness. Briefly, it first trains a classifier with the training samples. Then, the classifier is frozen and the confidence network (i.e., trustworthiness predictor) is trained (or fine-tuned) with the training samples. In general, we follows this learning scheme.
This work focuses on the trustworthiness on the predictions yielded by the publicly available pretrained classifiers, i.e., ViT [7] and ResNet [5]. We use the pre-trained backbones as the backbones of the oracles for general purposes. In this sense, the set of the oracle’s parameters can be split into two parts, one is related to the backbone and the other one is related to the head, i.e., θ = {θbackbone, θhead}. θbackbone is used to generated the intermediate feature xout and θhead = {w, b} are usually the weights of a linear function to generate the discriminative feature z = w>xout + b. With the classifier, the oracle, a pre-defined loss, and the training samples, we can optimize problem (3) to find the optimal parameters for the oracle.

heading: 3.2 Steep Slope Loss

text:
The conceptual workflow of the proposed steep slope loss is shown in Fig. 2a. The core idea is that we exploit the graph characteristics of the exponential function and the softsign function to establish two slides such that the features z w.r.t. the positive class ride down the positive slide to the right bottom and the features z w.r.t. the negative class ride down the negative slide to the left bottom. z is defined as the signed distance to the hyperplane (i.e., oracle head). Given an image x, z = fθ(x) can be broken down into
z = w>xout + b
‖w‖ , xout = fθbackbone(x). (7)
The signed distance to the hyperplane has a geometric interpretation: its sign indicates in which half-space xout is and its absolute value indicates how far xout is away from the hyperplane.
It is desired that the signed distance of xout with ground-truth label o = 1 (o = 0) tends towards +∞ (−∞) as much as possible. To this end, we define the steep slope (SS) loss function as follows
`SS(fθ(x), o) = o · ( exp ( α+z
1 + |z|
) − exp(−α+) ) ︸ ︷︷ ︸
Positive slide
+(1− o) · ( exp ( −α−z 1 + |z| ) − exp(−α−) ) ︸ ︷︷ ︸
Negative slide
(8)
where α+, α− ∈ R+ control the slope of the positive slide and the negative slide, respectively. If z w.r.t. the positive class is on the left-hand side of z = 0, minimizing the loss would push the point on the hill down to the bottom, i.e., the long tail region indicating the well-classified region. Similarly, z
w.r.t. the negative class would undergo a similar process. exp(−α+) and exp(−α−) are vertical shifts for the positive and negative slides such that `ss has a minimum value 0. Note that the proposed steep slope loss is in the range [0,maximum{exp(α+) − exp(−α+), exp(α−) − exp(−α−)}], whereas the cross entropy loss and the focal loss are in the range [0,+∞). The proposed steep slope loss can work with the output of the linear function as well. This is because the signed distance and the output of the linear function have a proportional relationship with each other, i.e., w
>xout+b ‖w‖ ∝ w >xout+ b.
Essentially, as shown in Fig. 2b, the cross entropy loss, focal loss, and steep slope loss work in a similar manner to encourage z w.r.t. the positive class to move the right-hand side of the positive threshold 0 and encourage z w.r.t. the negative class to move the left-hand side of the negative threshold 0, which is analogous to the sliding motion. The proposed steep slope loss is more tractable to control the steepness of slopes than the cross entropy loss and focal loss. This leads to an effective learning to yield discriminative feature for predicting trustworthiness.

heading: 3.3 Generalization Bound

text:
With the proposed steep slope loss, we are interested in the generalization bound of trustworthiness predictors. For simplicity, we simplify a trustworthiness predictor as f ∈ F , where F is a finite hypothesis set. The risk of predicting trustworthiness is defined asR(f) = E(x,y)∼P [`SS(f(x), o)], where P is the underlying joint distribution of (x, o). As P is inaccessible, a common practice is to use empirical risk minimization (ERM) to approximate the risk [19], i.e.,R̂D(f) = 1|D| ∑|D| i=1 `SS(f(xi), oi).
The following theorem provides an insight into the correlation between the generalization bound and the loss function in the learning task for predicting trustworthiness. Theorem 3.1. Denote maximum{exp(α+)− exp(−α+), exp(α−)− exp(−α−)} as `maxSS . `SS ∈ [0, `maxSS ]. Assume F is a finite hypothesis set, for any δ > 0, with probability at least 1 − δ, the following inequality holds for all f ∈ F:
|R(f)− R̂D(f)| ≤ `maxSS
√ log |F|+ log 2δ
2|D|
The proof sketch is similar to the generalization bound provided in [20] and the detailed proof is provided in the appendix A.
A desired characteristic of the proposed steep slope loss is that it is in a certain range determined by α+ and α−, as discussed in Section 3.2. This leads to the generalization bound shown in Theorem 3.1. The theorem implies that given a generic classifier, as the number of training samples increases, the empirical risk would be close to the true risk with a certain probability. On the other hand, the cross entropy loss, focal loss, and TCP loss are not capped in a certain range. They do not fit under Hoeffding’s inequality to derive the generalization bounds.

heading: 4 Related Work

text:
Loss Function. Loss function is the key to search for optimal parameters and has been extensively studied in a range of learning tasks [9, 18, 21, 22]. Specifically, the cross entropy loss may be the most widely-used loss for classification [21], however, it is not optimal for all cases. Lin et al. [18] propose the focal loss to down-weight the loss w.r.t. well-classified examples and focus on minimizing the loss w.r.t. misclassified examples by reshaping the cross entropy loss function. Liu et al. [22] introduce the large margin softmax loss that is based on cosine similarity between the weight vectors and the feature vectors. Nevertheless, it requires more hyperparameters and entangles with the linear function in the last layer of the network. Similar to the idea of focal loss, the proposed steep slope loss is flexible to reshape the function graphs to improve class imbalance problem. TCP’s confidence loss is used to train the confidence network for predicting trustworthiness [9]. We adopt TCP as a baseline.
Trustworthiness Prediction. Understanding trustworthiness of a classifier has been studied in the past decade [8, 9, 13, 23, 24]. Hendrycks and Gimpel [13] aim to detect the misclassified examples according to maximum class probability. This method depends on the ranking of confidence scores and is proved to be unreliable [8]. Monte Carlo dropout (MCDropout) intends to understand trustworthiness through the lens of uncertainty estimation [23]. However, it is difficult to distinguish
the trustworthy prediction from the incorrect but overconfident predictions [9]. Jiang et al. [8] propose a confidence measure named trust score, which is based on the distance between the classifier and a modified nearest-neighbor classifier on the test examples. The shortcomings of the method are the poor scalability and computationally expensive. Recently, instead of implementing a standalone trustworthiness predictor, Moon et al. [24] use the classifier’s weights to compute the correctness ranking. The correctness ranking is taken into account in the loss function such that the classification accuracy is improved. This method relies on pairs of training samples for computing the ranking. Moreover, it is unclear if the classifier improved by the correctness ranking outperforms the pretrained classifier when applying the method on large-scale datasets, e.g., ImageNet. In contrast, the learning scheme to train the trustworthiness predictor (i.e., confidence network) in [9] is standard and does not affect the classifiers. Its efficacy has been verified on small-scale datasets. In this work, we follows the learning scheme used in [9] and focus on predicting trustworthiness on complex dataset.
Imbalanced Classification. In classification task, the ratio of correct predictions to incorrect predictions is expected to be large due to the advance in deep learning methods. This is aligned with the nature of imbalanced classification [25, 26, 27, 28, 29], which generally employ re-sampling and re-weighting strategies to solve the problem. However, predicting trustworthiness is different from imbalanced classification as it is not aware of the visual concepts but the correctness of the predictions, whereas the classification task relies on the visual concepts. Specifically, the re-sampling based methods [25, 26, 29] may not be suitable for the problem of predicting trustworthiness. It is difficult to determine if a sample is under-represented (over-represented) and should be over-sampled (under-sampled). The re-weighting based methods [26, 28] can be applied to any generic loss functions, but they hinge on some hypothesis related to imbalanced data characteristics. For instance, the class-balanced loss [28] is based on the effective number w.r.t. a class, which presumes number of samples is known. However, the number of samples w.r.t. a class is not always assessible, e.g., in the online learning scheme [30, 31]. Instead of assuming some hypothesis, we follow the standard learning scheme for predicting trustworthiness [8, 9].

heading: 5 Experiment & Analysis

text:
In this section, we first introduce the experimental set-up. Then, we report the performances of baselines and the proposed steep slope loss on ImageNet, followed by comprehensive analyses.
Experimental Set-Up. We use ViT B/16 [7] and ResNet-50 [5] as the classifiers, and the respective backbones are used as the oracles’ backbones. We denote the combination of oracles and classifiers as 〈O, C〉. There are four combinations in total, i.e., 〈ViT, ViT〉, 〈ViT, RSN〉, 〈RSN, ViT〉, and 〈RSN, RSN〉, where RSN stands for ResNet. In this work, we adopt three baselines, i.e., the cross entropy loss [21], focal loss [18], and TCP confidence loss [9], for comparison purposes.
The experiment is conducted on ImageNet [11], which consists of 1.2 million labeled training images and 50000 labeled validation images. It has 1000 visual concepts. Similar to the learning scheme in [9], the oracle is trained with training samples and evaluated on the validation set. During the training process of the oracle, the classifier works in the evaluation mode so training the oracle would not affect the parameters of the classifier. Moreover, we conduct the analyses about how well the learned oracle generalizes to the images in the unseen domains. To this end, we apply the widely-used style transfer method [32] and the functional adversarial attack method [33] to generate two variants of the validation set, i.e., stylized validation set and adversarial validation set. Also, we adopt ImageNet-C [34] for evaluation, which is used for evaluating robustness to common corruptions.
The oracle’s backbone is initialized by the pre-trained classifier’s backbone and trained by fine-tuning using training samples and the trained classifier. Training the oracles with all the loss functions uses the same hyperparameters, such as learning rate, weight decay, momentum, batch size, etc. The details for the training process and the implementation are provided in Appendix B.
For the focal loss, we follow [18] to use γ = 2, which leads to the best performance for object detection. For the proposed loss, we use α+ = 1 and α− = 3 for the oracle that is based on ViT’s backbone, while we use α+ = 2 and α− = 5 for the oracle that is based on ResNet’s backbone.
Following [9], we use FPR-95%-TPR, AUPR-Error, AUPR-Success, and AUC as the metrics. FPR95%-TPR is the false positive rate (FPR) when true positive rate (TPR) is equal to 95%. AUPR is the area under the precision-recall curve. Specifically, AUPR-Success considers the correct prediction
as the positive class, whereas AUPR-Error considers the incorrect prediction as the positive class. AUC is the area under the receiver operating characteristic curve, which is the plot of TPR versus FPR. Moreover, we use TPR and true negative rate (TNR) as additional metrics because they assess overfitting issue, e.g., TPR=100% and TNR=0% imply that the trustworthiness predictor is prone to view all the incorrect predictions to be trustworthy.
Performance on Large-Scale Dataset. The result on ImageNet are reported in Table 1. We have two key observations. Firstly, training with the cross entropy loss, focal loss, and TCP confidence loss lead to overfitting the imbalanced training samples, i.e., the dominance of trustworthy predictions. Specifically, TPR is higher than 99% while TNR is less than 1% in all cases. Secondly, the performance of predicting trustworthiness is contingent on both the oracle and the classifier. When a high-performance model (i.e., ViT) is used as the oracle and a relatively low-performance model (i.e., ResNet) is used as the classifier, cross entropy loss and focal loss achieve higher TNRs than the loss functions with the other combinations. In contrast, the two losses with 〈ResNet, ViT〉 lead to the lowest TNRs (i.e., 0%).
Despite the combinations of oracles and classifiers, the proposed steep slope loss can achieve significantly higher TNRs than using the other loss functions, while it achieves desirable performance on FPR-95%-TPR, AUPR-Success, and AUC. This verifies that the proposed loss is effective to improve the generalizability for predicting trustworthiness. Note that the scores of AUPR-Error and TPR yielded by the proposed loss are lower than that of the other loss functions. Recall that AUPR-Error aims to inspect how easy to detect failures and depends on the negated trustworthiness confidences w.r.t. incorrect predictions [9]. The AUPR-Error correlates to TPR and TNR. When TPR is close to 100% and TNR is close to 0%, it indicates the oracle is prone to view all the predictions to be trustworthy. In other words, almost all the trustworthiness confidences are on the right-hand side of p(o = 1|θ,x) = 0.5. Consequently, when taking the incorrect prediction as the positive class, the negated confidences are smaller than -0.5. On the other hand, the oracle trained with the proposed loss intends to yield the ones w.r.t. incorrect predictions that are smaller than 0.5. In general, the negated confidences w.r.t. incorrect predictions are greater than the ones yielded by the other loss functions. In summary, a high TPR score and a low TNR score leads to a high AUPR-Error.
To intuitively understand the effects of all the loss functions, we plot the histograms of trustworthiness confidences w.r.t. true positive (TP), false positive (FP), true negative (TN), and false negative (FN) in Fig. 3. The result confirms that the oracles trained with the baseline loss functions are prone to predict overconfident trustworthiness for incorrect predictions, while the oracles trained with the proposed loss can properly predict trustworthiness for incorrect predictions.
Separability between Distributions of Correct Predictions and Incorrect Predictions. As observed in Fig. 3, the confidences w.r.t. correct and incorrect predictions follow Gaussian-like distributions. Hence, we can compute the separability between the distributions of correct and incorrect predictions from a probabilistic perspective. Given the distribution of correct predictions N1(µ1, σ21) and the distribution of correct predictions N2(µ2, σ22), we use the average Kullback–Leibler (KL) divergence d̄KL(N1,N2) [35] and Bhattacharyya distance dB(N1,N2) [36] to measure the separability. More details and the quantitative results are reported in Appendix G. In short, the proposed loss leads to larger separability than the baseline loss functions. This implies that the proposed loss is more effective to differentiate incorrect predictions from correct predictions.
Performance on Small-Scale Datasets. We also provide comparative experimental results on smallscale datasets, i.e., MNIST [10] and CIFAR-10 [37]. The results are reported in Table 2. The proposed loss outperforms TCP† on metric FPR-95%-TPR on both MNIST and CIFAR-10, and additionally achieved good performance on metrics AUPR-Error and TNR on CIFAR-10. This shows the proposed loss is able to adapt to relatively simple data. More details can be found in Appendix B.1.
Generalization to Unseen Domains. In practice, the oracle may run into the data in the domains that are different from the ones of training samples. Thus, it is interesting to find out how well the learned oracles generalize to the unseen domain data. Using the oracles trained with the ImageNet training set (i.e., the ones used in Table 1), we evaluate it on the stylized ImageNet validation set [32], adversarial ImageNet validation set [33], and corrupted ImageNet validation set [34]. 〈ViT, ViT〉 is used in the experiment.
The results on the stylized ImageNet, adversarial ImageNet, and ImageNet-C are reported in Table 3, More results on ImageNet-C are reported in Table A1. As all unseen domains are different from the
one of the training set, the classification accuracies are much lower than the ones in Table 1. The adversarial validation set is also more challenging than the stylized validation set and the corrupted validation set. As a result, the difficulty affects the scores across all metrics. The oracles trained with the baseline loss functions are still prone to recognize the incorrect prediction to be trustworthy. The proposed loss consistently improves the performance on FPR-95%-TPR, AUPR-Sucess, AUC, and TNR. Note that the adversarial perturbations are computed on the fly [33]. Instead of truncating the sensitive pixel values and saving into the images files, we follow the experimental settings in [33] to evaluate the oracles on the fly. Hence, the classification accuracies w.r.t. various loss function are slightly different but are stably around 6.14%.
Selective Risk Analysis. Risk-coverage curve is an important technique for analyzing trustworthiness through the lens of the rejection mechanism in the classification task [9, 38]. In the context of predicting trustworthiness, selective risk is the empirical loss that takes into account the decisions, i.e., to trust or not to trust the prediction. Correspondingly, coverage is the probability mass of the non-rejected region. As can see in Fig. 4a, the proposed loss leads to significantly lower risks, compared to the other loss functions. We present the risk-coverage curves w.r.t. all the combinations of oracles and classifiers in Appendix E. They consistently exhibit similar pattern.
Ablation Study. In contrast to the compared loss functions, the proposed loss has more hyperparameters to be determined, i.e., α+ and α−. As the proportion of correct predictions is usually larger than that of incorrect predictions, we would prioritize α− over α+ such that the oracle is able to recognize a certain amount of incorrect predictions. In other words, we first search for α− by freezing α+, and then freeze α− and search for α+. Fig. 4b and 4c show how the loss, TPR, and TNR vary with various α−. In this analysis, the combination 〈ViT, ViT〉 is used and α+ = 1. We can see that
α− = 3 achieves the optimal trade-off between TPR and TNR. We follow a similar search strategy to determine α+ = 2 and α− = 5 for training the oracle with ResNet backbone.
Effects of Using z = w>xout + b. Using the signed distance as z, i.e., z = w >xout+b ‖w‖ , has a geometric interpretation as shown in Fig. 2a. However, the main-stream models [5, 6, 7] use z = w>xout + b. Therefore, we provide the corresponding results in appendix F, which are generated by the proposed loss taking the output of the linear function as input. In comparison with the results of using z = w
>xout+b ‖w‖ , using z = w >xout + b yields comparable scores of FPR95%-TPR, AUPR-Error, AUPR-Success, and AUC. Also, TPR and TNR are moderately different between z = w
>xout+b ‖w‖ and z = w >xout + b, when α+ and α− are fixed. This implies that TPR and TNR are sensitive to ‖w‖. This is because the normalization by ‖w‖ would make z more dispersed in value than the variant without normalization. In other words, the normalization leads to long-tailed distributions while no normalization leads to short-tailed distributions. Given the same threshold, TNR (TPR) is determined by the location of the distribution of negative (positive) examples and the extent of short/long tails. Our analysis on the histograms generated with and without ‖w‖ normalization verifies this point.
Steep Slope Loss vs. Class-Balanced Loss. We compare the proposed loss to the class-balanced loss [28], which is based on a re-weighting strategy. The results are reported in Appendix H. Overall, the proposed loss outperforms the class-balanced loss, which implies that the imbalance characteristics of predicting trustworthiness is different from that of imbalanced data classification.

heading: 6 Conclusion

text: In this work, we study the problem of predicting trustworthiness on a large-scale dataset. We observe that the oracle, i.e., trustworthiness predictor, trained with the cross entropy loss, focal loss, and TCP confidence loss lean towards viewing incorrect predictions to be trustworthy due to overfitting. To improve the generalizability of the oracles, we propose the steep slope loss that encourages the features w.r.t. correct predictions and incorrect predictions to be separated from each other. We evaluate the proposed loss on ImageNet through the lens of the trustworthiness metrics, selective classification metric, and separability of distributions, respectively. Experimental results show that the proposed loss is effective in improving the generalizability of trustworthiness predictors.

heading: 7 Societal Impact

text:
Learning high-accuracy models is a long-standing goal in machine learning. Nevertheless, due to the complexity of real-world data, there is still a gap between state-of-the-art classification models and a perfect one. Hence, there is a critical need to understand the trustworthiness of classification models, i.e., differentiate correct predictions and incorrect predictions, in order to safely and effectively apply the models in real-world tasks. Models with the proposed loss achieve considerably better performance with various trustworthiness metrics. They also show generalizability with both indistribution and out-of-distribution images at a large scale. In addition, while existing trustworthiness models focus on a high TPR and tend to view all incorrect predictions to be trustworthy (i.e., TNR close to 0), false positives may lead to high consequent cost in some real-world scenarios such as critical applications in medicine (e.g., a false positive leading to unnecessary and invasive tests or treatment such as biopsies or surgery, or harmful side effects in medicine) and security (e.g., counter terrorism). It is thus of great importance for a trustworthiness model to flexibly trade off between TPR and TNR. To this end, the proposed loss allows the underlying distributions of positive and negative examples to be more separable, enabling a more effective trade-off between them.
Acknowledgments and Disclosure of Funding
This research was funded in part by the NSF under Grants 1908711, 1849107, and in part supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

title: Learning to Predict Trustworthiness with Steep Slope Loss