extfieldsofstudy:
Computer Science
Mathematics

text:
Dimensionality Reduction using Elastic Measures

With the recent surge in big data analytics for hyper-dimensional data there is a renewed interest in dimensionality reduction techniques for machine learning applications. In order for these methods to improve performance gains and understanding of the underlying data, a proper metric needs to be identified. This step is often overlooked and metrics are typically chosen without consideration of the underlying geometry of the data. In this paper, we present a method for incorporating elastic metrics into the t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP). We apply our method to functional data, which is uniquely characterized by rotations, parameterization, and scale. If these properties are ignored, they can lead to incorrect analysis and poor classification performance. Through our method we demonstrate improved performance on shape identification tasks for three benchmark data sets (MPEG-7, Car data set, and Plane data set of Thankoor), where we achieve 0.77, 0.95, and 1.00 F1 score, respectively.

Introduction
With the ever increasing number of observations and expanding number of variables in modern datasets, the use of dimensionality reduction as a first step for training machine learning models has gained a renewed interest. Methods such as the t-distributed Stochastic Neighbor Embedding (t-SNE) (van der Maaten and Hinton, 2008) and the Uniform Mani-fold Approximation and Projection (UMAP) (McInnes et al., 2020), have enabled improved classification performance over methods such as principal component analysis (PCA) and factor analysis. This is due to their ability to accurately embed complicated data structures into small dimensional spaces by computing the distances between observations in higher dimensional space. The choice of distance function is typically left to the user, where the default metric is the Euclidean distance, which is deficient in higher dimensional space. It has been shown that the UMAP is sensitive to the choice of distance metric and can effect the results significantly (McInnes et al., 2020).
When computing the distances between observations, a metric which is symmetric, isometric, and obeys the triangle inequality is preferred. These properties lead to pairwise distance computations that are independent of input order and can be degenerate for some data types (Srivastava et al., 2011b;Tucker et al., 2013). For example, the Euclidean distance is not a proper distance on functional data in R 1 under phase variation (Tucker et al., 2013), the space of open and closed curves (Srivastava and Klassen, 2016), and any trajectory on a Riemmanian manifold (Su et al., 2014).
In this paper, we demonstrate that selecting a distance metric that is consistent with the data topology drastically improves the performance of both t-SNE and UMAP on functional data. We apply this method to a clustering of curves tasks in R n , where we choose our distance metric such that it measures the shape of these curves by accounting for rotation, scale, and parameterization. We utilize the elastic shape framework (Srivastava and Klassen, 2016), which yields two metrics that properly measure distance in both shape and amplitude, and provides a distance from one shape to another in sampling variability.
When applied to the space of open curves, a Riemmanian manifold, we are able to obtain an initialization point for dimensionality reduction that is consistent with the properties of the data. We also apply this framework to any trajectory on a Riemmanian manifold using the square-root velocity transformation. This paper is organized as follows. In Section 2 we provide an overview of the Elastic Shape Analysis (ESA) framework, in Section 3 we review both t-SNE and UMAP, in Section 4 we provide results on two data sets, one being a set of shapes from the MPEG7, shapes of cars extracted from video, and fighter jet shapes, and finally, we provide a discussion and concluding remarks in Section 5.

Elastic Shape Analysis
Elastic Shape Analysis (ESA) is a collection of techniques for registering and analyzing functional data, using the process of phase-amplitude separation. From the separated components, statistical analysis can be performed on the individuals phase and amplitude components (Kurtek et al., 2011;Srivastava et al., 2011a;Tucker et al., 2013). In this section we provide a review of ESA and refer the reader to (Kurtek et al., 2011;Srivastava et al., 2011a;Srivastava and Klassen, 2016;Tucker et al., 2013) for a complete overview.
Phase and amplitude represent two orthogonal components of a function's shape, which are properties of a function that remain invariant to shape-preserving transformations (rotation, translation, scaling, and phase) (Dryden and Mardia, 2016;Srivastava and Klassen, 2016). The phase component represents the horizontal variability within trajectories or reparameterization, where as the amplitude component represents the vertical variability independent of phase. In order to analyze these components the need to be separated through alignment, Srivastava et al. (2011a) demonstrated that the Fisher-Rao metric is a proper distance and provides this separation.
To compute the alignment using the Fisher-Rao metric, the Square Root Velocity Function (SRVF) (Srivastava et al., 2011a) is used for registration of functions in R n . For real-valued curves, the SRVF performs a bijective mapping of the real-valued curve f to its normalized gradient f / |f |. The registration of two real-valued curves is performed by elastically deforming the domain of one function such that the L 2 distance between the SRVFs of the two curves is minimized. The amount of elastic deformation required is measured by the phase distance (Section 2.2), while the residual distance between the SRVF, post registration (rotation, and scaling), defines the amplitude distance between them (Section 2.1). Together they are known as elastic distances. By registering SRVFs instead of trajectories directly, the phase and amplitude distances become proper metrics and are invariant to shape-preserving transformations.

Amplitude distance for R n valued functions
Let F R n = {f : [0, 1] → R n , f differentiable} be the class of differentiable trajectories on [0, 1] mapping to R n with n ≥ 2 1 . In higher dimensional Euclidean space (n ≥ 2), the scale, rotation, and phase of a trajectory need to be accounted for in order to isolate its shape. Scale variability is removed by standardizing each trajectory to have unit length, i.e., each trajectory is divided by the L 2 norm of its gradient: Rotation variability is accounted for using the space of rotation matrices, the Special Orthogonal Group SO(n) and is defined as the group of orthogonal matrices with a determinant of one. For trajectories in R 2 , SO(2) defines the rotations around a point and in R 3 SO(3) defines rotations around a line. The action of SO(n) on a trajectory f is denoted as O(f ) and is defined pointwise as where Of (t) represents a standard matrix multiplication of the n × n matrix O and the n × 1 vector f (t). See Vladimir (1994) for more details on the Orthogonal groups and their properties. The optimal rotation matrix in SO(n) is found alongside the optimal reparameterization when computing the amplitude distance.
Phase variability in R n is represented using the space Γ = {γ : I → I|{γ, γ −1 } ∈ C 1 (I),γ > 0} of all positive slope diffeomorphisms of the unit interval. Together with the operation of composition, Γ is a group and is defined with respect to the domain I = [0, 1] . The reparameterization of a curve f by a warping function γ ∈ Γ is denoted as the operation f • γ.
We desire a distance that is invariant to simultaneous reparameterizations, which allows us to use the SRVF as the following transformation on trajectories f ∈ F n R : The SRVF is defined as the following transformation on trajectories f ∈ F n R : where |f (t)| is the absolute value of f at t. This transformation maps trajectories onto the L 2 ball of radius one and is a bijective mapping, up to an additive constant (Srivastava and Klassen, 2016).
Since the space of SRVFs is the L 2 ball then the norm on the SRVFs is the arc length distance between points on the sphere: where q f = SRV F (f ) and q g = SRV F (g) for two trajectories f, g ∈ F R n and q f (t), q g (t) denotes the inner product of the vectors q f (t), q g (t). To convert this into an amplitude distance we need to place f and g in phase and rotation with each other. This optimization is summarized as the following amplitude distance on R n valued trajectories (Srivastava et al., 2011a): where q f and q O(g•γ) denote the SRVF's of f and O(g • γ) respectively.

Phase Space
We will define the phase space using diffeomorphic mapping. The phase space of the unit interval [0, 1] as defined above is This diffeomorphic constraint gives rise to the notion of elasticity because the elements of Γ, i.e., phase functions, can only smoothly stretch and contract portions of the unit interval so that it maps back to itself. Phase is generally thought of as the representation of a trajectory because any trajectory with domain [0, 1] can be warped by a phase function to appear differently. The amplitude will be taken to be those features of a trajectory that remain unchanged under any possible warping.

Phase distance
To define phase distance we use the optimal γ that defines the amplitude distance. The phase space Γ is a nonlinear manifold with no known geometry so we use the SRVF to map Γ to a known geometry (Tucker et al., 2013). Phase functions are positive for all t ∈ [0, 1] and ||q γ || = 1 0 γ (t) 2 dt = 1, so the SRVF maps Γ onto the positive orthant of a unit Hilbert Sphere. Thus the phase distance is defined as where ψ = √γ and ·, · is the inner product between the two vectors (Srivastava and Klassen, 2016). The metric d p (γ 1 , γ 2 ) is essentially measuring the amount of elastic deformation needed to compare the shapes of f 1 and f 2 , when γ 1 = id is the identity warping.

Dimension Reduction
In dimensionality reduction, we are interested in finding a low dimensional embedding space, Y , for the high dimensional space, X, such that Y is as similar to X as possible and that the representation of the reduced space facilitates better data analysis (e.g., classification with good class separation). For the dimensionatliy reduction to work we want to define the similarities between two objects x i and x j in X and their mapping to Y . Recently, two methods have become highly utilized in the literature for performing dimensionality reduction; t-SNE and UMAP. In the following sections we will give a brief overview of the two methods.

t-SNE: t-distributed Stochastic Neighbor Embedding
A t-distributed Stochastic Neighbor Embedding (t-SNE) (van der Maaten and Hinton, 2008) embedding defines probabilities proportional to the similarity of the objects x i and x j , and are defined as is the Euclidean distance between the two objects (as defined in (van der Maaten and Hinton, 2008)). The similarity between the two objects is the conditional probability that x i is similar to x j in proportion to the probability density under a Gaussian distribution centered at x i .
The total probability or similarity is defined as where t-SNE aims to learn a d-dimensional mapping, Y , that reflects the similarities p ij .
The similarities between y i and y j are defined as: for which the student t-distribution measures these similarities in the mapping space.
The locations of the points (y i ) in the lower dimensional space is determined by minimizing the Kullback-Leibler divergence of the distribution P from the distribution Q, i.e., This minimization is performed using gradient descent and is the mapping between X and Y .

UMAP: Uniform Manifold Approximation and Projection
Uniform Manifold Approximation and Projection (UMAP) approximates a manifold by constructing a fuzzy simplicial set representation, which is performed on high dimensional data (X) and on a low dimensional representation (Y ∈ R d ). The selected representation is the one that optimizes the cross entropy between the high and low dimensional spaces.
UMAP is similar to the approach of t-SNE, which constructs a probability distribution over pairs of high-dimensional objects. In t-SNE similar objects have a high probability of being picked, whereas dissimilar points have an extremely small probability of being picked.
t-SNE defines a similar probability distribution over the points in the low-dimensional map, and minimizes the Kullback-Leibler divergence between the two distributions.
In UMAP, the high dimensional similarities, v j|i , are local fuzzy simplicial set memberships based on smooth nearest-neighbor (NN) distances v j|i from x i ∈ X to one of its k is the Euclidean distance on the learned manifold. The parameter ρ i is the distance to nearest neighbor and σ i is a normalizing constant. The symmetrization are used to produce an undirected graph structure representing the 1-dimensional frame of the fuzzy simplicial set, is carried out by fuzzy set union using the probabilistic t-conorm The graph defined by the v ij is then embedded into a low dimensional space Y , where the dimension is prescribed as a parameter. The low dimensional similarities between the projections y i and y j , of x i and x j , into Y via the initial embedding are given by where a and b are user defined and a gradient descent procedure is used to find them. The defaults for UMAP are a ≈ 1.929 and b ≈ 0.7915 (McInnes et al., 2020).
The cost function that is optimized to find the embedding is which penalizes discrepancies in the relative distributions of similarities in X and Y .

Elastic Modifications
To perform dimensionality reduction, it is critical that the distance, d(x i , x j ), is appropriate for the underlying data. In this work, we perform dimensionality reduction on curves that lie in R n . We then can utilize the distance defined in Equation 3  We give a brief review of the geometry of the space of PDFs but refer the reader to Kurtek and Bharath (2015) for a more in-depth review. We focus here on univariate densities on [0, 1], but the theory can easily be extend for all finite dimensional densities.
Let P = {p : [0, 1] → R ≥0 | 1 0 p(x) dx = 1} as the space of all PDFs and we define the tangent space T p (P) = {d p : [0, 1] → R| 1 0 d p (x)p(x) dx = 0}. The inner-product on this space, known as the Fisher-Rao Riemmanian metric (Kass and Vos, 1997;Rao, 1945) is given by: Cencov (1982) showed that this metric is invariant to re-parameterizations. However, the metric is difficult to compute, requiring numerical methods and large computations. Bhattacharya (1943) proposed a convenient square-root transformation with simplifies the computation of the metric and is the transformed space where the Hellinger distance is an extrinsic metric. Define the transformation φ(p) = √ p as the square-root density of PDF p, which is similar to SRVF described previously. The corresponding space of SRDs is which the positive orthant of the Hilbert Sphere. Since the geometry is known in this transformed space we can define the geodesic distance between two PDFs p 1 , p 2 ∈ P under the Fisher-Rao metric using the SRDS √ p 1 , √ p 2 ∈ Ψ is the great circle defined as Therefore, we can replace the Kullback-Leibler divergence in t-SNE with a proper metric on the space of PDFs using Equation 5 similar to divergences done in Bunte et al. (2012); Im et al. (2018). The use of an intrinsic metric is more natural than an extrinsic method as it is the exact distance between two PDFs and is invariant to re-parametrizations of the PDFs We will denote this modification to t-SNE as elastic t-SNE or et-SNE and will compare t-SNE, UMAP, and elastic t-SNE using the elastic metrics on the space of curves in R n .

Results
In this section we provide results on two data set. The first data set will be analyzing a set of shapes where we have 65 shape classes from the MPEG-7 Core Experiment CE-Shape-1 Test Set (Jeannin and Bober, 1999). This data set is used for benchmarking shape matching algorithms and includes binary images grouped into categories by their content, not their appearance or shape. The second is the fighter plane shapes and the car shapes from Thakoor et al. (2007) and includes part of the MPEG-7 data set.

MPEG7
In this section, we classify real world curve data taken from the MPEG-7 database (Jeannin and Bober, 1999). The full database has 1300 shape samples, 20 shapes for each class, and 65 shape classes. An example of some of the shapes from the MPEG-7 bases is presented in Figure 1. Note the differences in rotations and scales of each of the shapes which complicates the analysis. Using the shape distance in Definition 3 we can compute the pairwise distance matrix between all shapes in the data set. Figure 2a provides the distance matrix using the elastic distance and for comparison we computed the distance metric using the Euclidean distance and is provided in Figure 2b. There is a definite difference in structure and visually there is a noticeable large amount of clustering in the elastic distance where the Euclidean distance is only showing roughly three large groups.  We computed both t-SNE, e-tSNE, and UMAP projecting the data onto two components using the elastic distance and the Euclidean distance for comparison. Figure 3 provides the t-SNE results with panel (a) using the elastic distance and panel (b) using the Euclidean distance. There is a distinct clustering of each of the classes and separability of the classes versus the Euclidean distance. Figure 4 provides the elastic t-SNE results with panel (a) using the elastic distance and panel (b) using the Euclidean distance. Comparing to standard t-SNE using the Kullback-Leibler divergence we see a slightly better separation of classes and as mentioned in above we see neighborhood sizes are more preserved. To add a more complete picture to the embedding we trained a Random Forest classifier (Breiman, 2001) using 5-fold cross-validation. Since, t-SNE was not originally defined for out-of-sample data we will focus on UMAP. In the following section, for the full data analysis we will utilize parametric t-SNE (Maaten, 2009;Poličar et al., 2021) which allows for outof-sample data. Figure 6 provides the average F1 score across folds with the standard deviation shaded, using the elastic metric and the Euclidean metric. Overall, for each class the F1 score is larger for each class and maintains on average over 0.8 while on average of the Euclidean is 0.6. Figure 7 provides the confidence matrix for the elastic metric in panel (a) and the Euclidean metric in panel (b). As was noted from the F1 score, the elastic distance for most of the classes does extremely well with few classes being misspecified.   The Euclidean distance does have trouble with a few of the classes with more than a few miss classifications.

Shape Database
In this section, we look at the overall classification performance utilizing UMAP and parametric t-SNE as a dimensionality reduction tool and then performing classification using a Random Forest classifier. Again we will compare the results against using the standard Euclidean metric in both dimension reduction techniques. For this study we will utilize the fighter jet shapes, car shapes, and the full MPEG7 database described and analyzed in the previous section.
The fighter airplane shape database includes Mirage,Eurofighter,Harrier, shapes. Since the F-14 has two possible shapes, one when its wings are closed and another when its wings are opened, the total number of shape classes is seven, where there are 30 shapes for each class for a total of 210 examples. The database was created from digital pictures of die-cast replica of the airplanes and were segmented to extract the contour.
The car shapes were generated using the method described in Thakoor and Gao (2005) where contours of vehicles were extracted from video clips. There are four classes in the group: sedan, pickup, minivan, and SUV and each have 30 samples for each class. The data set shows larger within-class variation, as shapes of vehicles of different makes and models vary and some contours are distorted due to the shadow also being extracted. This data set contains more real world variability in applying shape classification metrics. Figure 8 presents the F1 score for each class for the plane data set in panel a) and the car data set in panel b). A Random Forest classifier was trained from the UMAP embedding using the designated distance. The shaded region shows the variability across the 5-fold cross-validation. In both data sets the elastic metric vastly outperforms the the Euclidean metric for all of the classes and shows less variability across the folds. The increase in performance is directly related to utilizing a proper distance and a metric that is invariant to scale, rotation, and re-parameterization of the shape. Being invariant to these actions is critical in any shape analysis/recognition task.     Lastly, to examine performance across multiple classifiers we compared the performance of the Random Forest (RF) against the Gradient Boosted Trees (GBT) (Friedman, 2001) and K-nearest neighbors classifiers (KNN) (Cover and Hart, 1967). The purpose of this comparison is to demonstrate that the method is independent of the classifier and to show the power is in choosing the proper metric for the geometry of the data under consideration. Table 2 presents the class averaged F1 score and MCC for the three classifiers for the MPEG-7 data set. For all three classifiers under both UMAP and t-SNE we see an improvement in classification performance when using the elastic metric over the Euclidean.
Across the classifiers, UMAP has the most consistent performance, while t-SNE obtains the highest under KNN, the performance under the other classifiers is quite reduced. The increase in performance between the two metrics is attributed to picking a metric that is a proper distance for the data being classified.

Discussion
We have proposed a new flexible elastic metric for functional data utilizing state-ofthe art dimensionality reduction methods. We have demonstrated its advantages over the current methods using current shape analysis benchmarking data set. Unlike the current cross-sectional metrics the elastic method accurately estimates the mean of the underlying data generating mechanism and handles rotation and parametrization as a nuisance accordingly. It is imperative to choose a metric that is respective of the underlying variability of the data and is respective of the geometry. Doing so will give advantages in multiple machine learning tasks. We have presented the metric for open and closed shapes in R n , but equivalent metrics exists for functions in R 1 (Tucker et al., 2013), surfaces (Kurtek et al., 2017), and images (Xie and Srivastava, 2016


extfieldsofstudy:
Computer Science
Engineering

text:
Too Fine or Too Coarse? The Goldilocks Composition of Data Complexity for Robust Left-Right Eye-Tracking Classifiers

The differences in distributional patterns between benchmark data and real-world data have been one of the main challenges of using electroencephalogram (EEG) signals for eye-tracking (ET) classification. Therefore, increasing the robustness of machine learning models in predicting eye-tracking positions from EEG data is integral for both research and consumer use. Previously, we compared the performance of classifiers trained solely on finer-grain data to those trained solely on coarse-grain. Results indicated that despite the overall improvement in robustness, the performance of the fine-grain trained models decreased, compared to coarse-grain trained models, when the testing and training set contained the same distributional patterns \cite{vectorbased}. This paper aims to address this case by training models using datasets of mixed data complexity to determine the ideal distribution of fine- and coarse-grain data. We train machine learning models utilizing a mixed dataset composed of both fine- and coarse-grain data and then compare the accuracies to models trained using solely fine- or coarse-grain data. For our purposes, finer-grain data refers to data collected using more complex methods whereas coarser-grain data refers to data collected using more simple methods. We apply covariate distributional shifts to test for the susceptibility of each training set. Our results indicated that the optimal training dataset for EEG-ET classification is not composed of solely fine- or coarse-grain data, but rather a mix of the two, leaning towards finer-grain.

INTRODUCTION
Recently, machine-learning classifiers have found more and more consumer applications as well as "on the field" usage. For example, machine learning models can be used to identify shopping motives relatively early in the search process [19], to detect workload strain in truck drivers [13], and to assess the diagnosis of many neurological diseases such as Autism Spectrum Disorder and Alzheimer's [4,11,30,31]. Machine learning approaches have also shown adequate performance on computer vision bioinformatics applications, medical image analysis, Eye-Tracking (ET) analysis, and Electroencephalography (EEG) analysis [1, 3, 8, 15, 22, 23, 25-27, 29, 36, 37, 40].
As machine learning classification is used in more and more unfamiliar environments, it is increasingly important for these classifiers to be robust. In the EEG-ET research, recent work in this area has focused on determining what machine learning models are best equipped to predict eye position from EEG signals [15] as well as eliminating the noise associated with EEG data collection automatically [17,21,24,28,33,39].
These approaches all examine robustness across a variety of underlying factors of data analysis. In this paper, We focus on the inherent distributional patterns and the underlying differences between finer-grain and coarser-grain data. For this paper, finer-grain data refers to data collected in a more complicated framework in an environment with more uncontrolled conditions. On the contrary, coarser-grain data refers to data collected in a more simplified collection format with more restrictions on the experiment's environment for the same task.
In medical research, fine-grain data collection methods have been explored with great success [9,20]. That is the use of data from more complex, or finer-grained, data collection methods to test for simpler tasks. Oftentimes, systems developed in a coarsegrained "lab conditions" only work in controlled environments, causing difficulties when utilized in uncontrolled conditions [16,34]. Recently, fine-grain data approaches have shown successful results for Covid-19 contact tracing machine learning classifiers [7]. In this paper, We want to emphasize the impact of data granularity as well as the beneficial effects of using fine-grain data in machine learning classification.
Previously, we showed that Machine Learning models trained on fine-grained data are more robust, meaning that they maintain more LG Tasks (Right) [12] consistent accuracies, than models trained on coarse-grained data when tested on data of different complexities [35]. We extend upon our previous findings by examining the idea of training on mixed fine-and coarse-grain datasets. To our knowledge, this approach was not tested with machine learning classifiers before. The goal of mixing datasets of varying data granularity is to improve upon the draw-back we previously found with training on purely finegrain data. That is the model's performance when tested on data of similar complexity drops a significant amount when compared to purely coarse-grain data. We suspect that a mixed complexity training set may perform adequately across a broader range of testing distributional patterns, including different and similar complexities.
In this study, we train machine learning models for left-right eyetracking classification using data from a mix of binary-classified (coarse-grain) and vector-based (fine-grain) collection frameworks. We then compare the results to models trained exclusively on either binary-classified or vector-based data. The goal is to expand upon our previous conclusions regarding the optimal method to increase both accuracy and robustness of machine learning classifiers using fine-and coarse-grain data. Robustness is determined by the accuracy after a covariate distributional shift. The distributional shift in combination with the different mixes of data complexity attempts to mimic realistic data which often contains varying distributional patterns. Since we had previously determined the superiority of fine-grain data in terms of robustness, we hypothesize that classifiers trained using more fine-grain data will attain higher accuracies after covariate distributional shifts are applied.
The purpose is to determine the "Goldilocks" or optimal training set composition of fine-and coarse-grain data for left-right gaze classification in terms of both accuracy and robustness as well as verify that fine-grain data performs better than coarse-grain data using a more encompassing experimental design.

DATA
We used the EEGEyeNet dataset due to its large size [12]. The dataset contains simultaneously-recorded EEG signals and Eye-Tracking data from three different experimental tasks. A summary table for the three tasks' metadata is in Appendix D. In this study, we used data from two experiments: pro-antisaccade (PA) and Large Grid (LG). In the PA trials, participants were asked to focus on a cue that appears on either the screen's center, horizontally left, or horizontally right, as shown in the left screen in Figure 1. Gaze positions in PA were restricted to the horizontal axis and were binary-classified either left or right, relative to the screen's center. In the LG trials, participants were asked to fixate on dots presented one at a time at 26 different screen positions, as shown in the right  Figure 1. Therefore, LG's framework enabled finer-grain data collection since it allowed more freedom for eye movements, and gaze positions were encoded in a two-dimensional format including both angle and amplitude, compared to the one-liner binary-encoded eye movements in PA.

EXPERIMENT DESIGN
The learning objective of the models trained in our experiment was to use EEG brain signals to predict the direction of a subject's gaze along the horizontal axis (whether they are looking to the left or the right). Although predictions for this task using the same dataset were previously made, they were performed exclusively using data from the PA tasks for training and testing [12].
In this paper, we train 13 classifiers using data that is composed of a mixture of both PA and LG data and perform a covariate distributional shift by comparing their performance based on their accuracy when tested on PA and LG data. The different compositions are described in Figure 2

Data Processing
Given the classification nature of our learning problem, data from the LG task, encoded as Angle and Amplitude, should be transformed and relabeled into the expected format as left or right for training left-right classification models. The convention we used for the relabelling process was that for angle ; when | | < 2 the data is classified as right, otherwise the data is classified as left. This logic was confirmed by the dataset's authors in Appendix A and is shown in Figure 3.

Models Training
As per EEGEyeNet authors' recommendation, we used the minimally preprocessed EEG data. Data processing was done using the NumPy library, and the model implementations were installed from the SKlearn library [18]. From a broader perspective, models were   LG PA Mixed  LG PA Mixed (60- 40) LG PA Mixed (80-20) LG PA trained and tested on 4 different combinations: models trained on mixed data and tested on PA data, models trained on mixed data and tested on LG data, models trained on PA data and tested on mixed data, and models trained on LG data and tested on mixed data. For each of the 4 combinations, the mixed data had 6 different compositions of PA and LG data, shown in Figure 2, which lead to 24 different combinations. However, 4 of these combinations are counted twice, leading to only 20 different combinations. A summary of the 20 unique combinations is shown in Table 1.

MODELS 4.1 Machine Learning Models
In this study, machine learning models operate on features extracted from the data rather than the data itself. Feature extraction has been applied in two steps. First, [12] applied a band-pass filter in frequencies in the range [8 -13 HZ] on the acquired signals through all trials. This choice of frequencies is based on suggestions from [6]. Following the filtering step, the Hilbert transform was applied, resulting in a complex time series from which targeted features were extracted for learning models. Since we are considering a classification problem, we experimented with classification-only models and models that can be applied to both classification and regression problems.

Linear Classifiers:
Linear classifiers gather discriminant classifiers that use linear decision boundaries between the feature vectors of each class [15]. In this paper, we will use Linear and Radial-Basis-function Support Vector Classifiers (SVC) and Normal and Shrinkage Linear Discriminant Analysis (LDA) classifiers. All four algorithms still perform well after several decades since they were first implemented in this field, with SVC outperforming other classifiers, especially for two-classes problems [2,14,15].

Ensemble Classifiers.
Ensemble (or voting) classifier is a machine learning classification algorithm that trains with different classification models and makes predictions through ensembling their predictions to make a stronger classification. These algorithms have been the gold standard for several EEG-based classification experiments [25]. In our study, we used Random Forest, XGBoost, GradientBoost, RUSBoost, and AdaBoost.

Naive Bayes and Decision Tree Classifiers. Naive Bayes (NB)
Classifier is the statistical Bayesian classifier [5]. It assumes that all variables are mutually correlated and contribute to some degree towards classification. It is based on the Bayes' Theorem and is commonly used with high dimensional inputs. On the other hand, a decision tree is not a statistically based one; rather, it is a data mining induction technique that recursively partitions the dataset using a depth-first greedy algorithm until all data is classified with a particular class. Both NB and the decision tree are relatively fast and well suited for large data. Furthermore, they can deal with noisy data, which makes them well suited for EEG classification applications [10].

Deep Learning Models
Deep learning is a subfield of machine learning algorithms in which computational models learn features from hierarchical representations of input data through successive non-linear transformations [29]. Deep learning methods, especially Convolutional Neural Network (CNN), performed well in several previous EEG band power (feature) based research [25]. Still, these methods have not demonstrated convincing and consistent improvements [15]. Given so and the expected high run time for such algorithms due to the dataset's large size, we only included a simple and relatively fast multi-layer perception neural network (MLP) in our experiment.

RESULTS
We trained and tested the machine learning models using datasets composed of both fine-and coarse-grain data (pro-antisaccade and "transformed" large grid) and compared the results. Thus, we had    20 combinations of training and testing datasets described in Table  1.
Determining the Goldilocks composition of fine-and coarsegrain data for training is the main objective of this paper as it will indicate the optimal method of maintaining high accuracy and consistency. We also verified the results of fine-versus coarse-grain data by finding the accuracies of models when trained on PA/LG and tested on mixed data.
Results regarding the accuracies of select models as well as the average accuracy for each combination are provided in Tables 2, 3, 4, and 5. Table 6 is a summary of Tables 2 and 3.
The average accuracy of models trained on PA and tested on mixed is 90.2% as shown by the average of the averages in Table 4. The average accuracy of models trained on LG and tested on mixed is 91.3% as shown by the average of the averages in Table 5. The accuracy of models train on LG is also more compact and closer to the average accuracy. Clearly, finer-grain data is in general more applicable to real life data as it is more accurate across a broader range of distributional patterns. This confirms our previous findings across a more complete range of data complexity.
Note that although we did not include standard deviations in our tables, we did calculate them, but since they were insignificant (generally less than 0.1%) we decided to not include them in our results.  Table 6 describes the average accuracy from 13 different machine learning model for left-right ET classification trained on data of different data complexity compositions. This paper utilizes the benchmark data from the EEGEyeNet dataset and 13 machine learning models to create left-right classifiers trained on data of varying compositions of fine-and coarsegrain data. The accuracies of the models are then tested using two testing sets with distinctly different distributional patterns (PA and LG). In this way, the effects of covariate distributional shifts are more apparent and provide insight on the optimal data complexity composition of fine-and coarse-grain data in order to reduce the impact of such phenomenon. The results also provide useful information on the types of machine learning classifiers that should be used for EEG-ET classification tasks.

Finding the Goldilocks Composition
Previously, we had concluded that machine learning classifiers trained purely on fine-grain data outperform machine learning classifiers trained purely on coarse-grain data in terms of robustness. This was confirmed by Tables 4 and 5. Therefore, we theorized that the ideal composition of data complexity would lean towards finergrain data.
The hypothesis is confirmed by Table 6. The composition of 20% coarse-grain data and 80% fine-grain data produced the most consistently accurate results as shown by the average percentages. This suggests that the optimal training set for EEG-ET classification should not be created using data of solely one distributional complexity, but rather a mix of both fine-and coarse-grain data, leaning towards more fine-grain data.

Determining the Best Classifiers for EEG-ET Classification
The consistently most accurate classifiers across Tables 4, 5, 2, and  3 are XGBoost, GradientBoost, and RandomForest. Alternatively, the consistently worst performing classifiers were RBF SVC and GaussianNB. Therefore regardless of distributional complexity, XG-Boost, GradientBoost, and RandomForest should be used to classify EEG-ET data.

Future Recommendations and Improvements
To further advance the work provided in this study, four steps are highlighted for future exploration. Although deep learning models were excluded due to inconsistent results [15], the main reason was due to restraints in time and resources. As deep learning models, especially CNN and Attention [32,38], have shown promising results in regards to EEG-ET classification, it is important to thoroughly explore the effects of fineversus coarse-grain data on the robustness of such models.
Additionally, the angle value is currently the only indicator used to determine whether the saccade is towards the left or the right. The amplitude was completely ignored in data processing. The amplitude could be incorporated by using it as a weighting/scaling factor to indicate the left-right extension and solve this as a regression problem. Based on the predictions made by the regression value, we can then classify it as either left or right utilizing finergrain data.
Furthermore, although we determined an approximately optimal training set, we are almost certain this is not the Goldilocks composition for mixed data complexity. Perhaps an application of gradient descent would be able to more precisely compute such a composition for each machine learning model and specific machine learning task. Additionally due to time restraints, we were unable to test our machine learning models trained on mixed distributional complexity against a broader range of testing sets like we did for exclusively fine-or exclusively coarse-grain trained classifiers in Tables 4 and 5.
Although our work discusses EEG-ET classification, the results can be potentially relevant to other machine learning models in other applications. We highly encourage further investigation on training machine learning models on data composed of different granularities across an array of other research topics.

CONCLUSION
The motivation behind this work was to extend upon our previous work on determining whether training machine learning models on finer-grain data leads to more robust models as well as to analyze the effects of training machine learning classifiers on datasets composed of mixed data complexity. We verified our previous findings across a broader range of distributional patterns as well as determined that ensemble methods are the most suitable machine learning classifiers for EEG-ET classification tasks. Furthermore we identified that EEG-ET machine learning classifiers seem to produce the most consistent results when the training set contains a mix of distributional patterns, leaning towards finer-grain data. We hope that future applications of practical EEG-ET interfaces utilize data of varying distributional patterns to increase classification accuracy and robustness.

APPENDICES 9.1 Appendix A
After emailing Ard Kastrati, Ard verified that for angle in radians: = 0 is right = 2 is down = is left = − 2 is up

Appendix B
The models were trained and tested on this environment settings: The total space occupied by the dataset on the device is 69.0574 GB, and the total time for training and testing was 30 mins on average.

Appendix C
The code for data processing and evaluation is provided here: https: //github.com/ayahia1/KDD-ML

Appendix D
The main reason for using EEGEyeNet in our study was its relatively large size. The table below presents the number of subjects, the number of sample data points, and the length of the recording time for each of the three experimental tasks in the EEGEyeNet dataset. The numbers below should prove the large size of this dataset.
Notes: First, in the table, experimental tasks are referred to as "paradigms. " Secondly, although the total number of subjects in the table is 486, some performed more than one experiment and, thus, referenced twice. The total number of unique subjects, when the dataset paper was first published, was 356. Finally, the recording time column contains the numbers rounded to the nearest hour.


extfieldsofstudy:
Medicine
Engineering
Computer Science

text:
Unsupervised Domain Adaptation for Segmentation with Black-box Source Model

Unsupervised domain adaptation (UDA) has been widely used to transfer knowledge from a labeled source domain to an unlabeled target domain to counter the difficulty of labeling in a new domain. The training of conventional solutions usually relies on the existence of both source and target domain data. However, privacy of the large-scale and well-labeled data in the source domain and trained model parameters can become the major concern of cross center/domain collaborations. In this work, to address this, we propose a practical solution to UDA for segmentation with a black-box segmentation model trained in the source domain only, rather than original source data or a white-box source model. Specifically, we resort to a knowledge distillation scheme with exponential mixup decay (EMD) to gradually learn target-specific representations. In addition, unsupervised entropy minimization is further applied to regularization of the target domain confidence. We evaluated our framework on the BraTS 2018 database, achieving performance on par with white-box source model adaptation approaches.

INTRODUCTION
Accurate tumor segmentation is an important prerequisite for early diagnosis and treatment planning, which has been substantially improved with advances in deep learning approaches. The performance of a pre-trained model, however, can be severely degraded, when a source distribution differs from a target distribution. For example, tumors with different grades are likely to yield different distributions, which hampers the application of a single segmentation model in segmenting tumors of varying degrees of severity and growth. 1 In addition, it is costly to annotate high-quality labeled data in the new target domain. Unsupervised domain adaptation (UDA) has been proposed to address the problem of domain shift. 2 In UDA, a segmentation model is typically trained using both source and target data, but only the source data are labeled.
Although UDA offers a promising solution to the problem of domain shift, due to the concerns over leakage of sensitive information contained in the patient data, it is often challenging to access source data, when deploying the developed model in the target domain. 3 To address this, Liu et al. proposed a source-free or source-relaxed UDA (i.e., white-box domain adaptation) for segmentation. 3 In that work, an off-the-shelf segmentation model was adapted to a target domain via a pre-trained model in a source domain, by transferring its batch normalization statistics. Recently, a deep inversion technique, 4 however, has shown that original training data can be recovered from knowledge used during white-box domain adaptation, which may leak confidential information. 5 Accordingly, this work aims to address the limitation, by developing a black-box domain adaptation approach that further restricts the use of knowledge from a source segmentation model.
To our knowledge, this is the first attempt at achieving UDA for a deep segmentation network using blackbox domain adaptation. The black-box setting provides a more effective way to protect privacy, compared with white-box domain adaptation approaches 3 or conventional UDA approaches. 6 Recently, Zhang et al. 5 proposed to use the black-box UDA for classification, with the class-wise noise rate estimation and category-wise sampling. However, that work is not applicable to the segmentation task to carry out pixel-wise classification. In previous works, segmentation based on the black-box domain adaptation was largely ignored, although it could be used in a more realistic and challenging scenario.

RELATED WORK
Unsupervised domain adaptation 2, 7-9 has been an important technology to address the difficulties such as domain shift and costly labeling in the new domains. Conventional approaches utilized both source and target domain data for training. 1,[10][11][12][13] Recently, source-free UDA [14][15][16] has been proposed to only use the pre-trained model rather than co-training the network with source and target domain data. A recent study 17 explored the shared or domain-specific batch-normalization statistics to achieve domain alignment. We note that domain generalization, 18 a closely related but different task, assumes that there are no target domain data in its training.
In this work, we target to tackle a more restrictive scenario, in which we only use a black-box pre-trained model, and do not rely on the network parameters. As stated above, a deep inversion technique, 4 has shown that original training data can be recovered from knowledge during white-box domain adaptation. 5 A contemporary work 5 proposed to adapt a black-box classification model to a target domain without both source domain data and the model structure. That work proposed iterative learning with noisy labels, which regarded the blackbox predictions as noisy labels. To the best of our knowledge, black-box UDA for segmentation has not been investigated. The semantic segmentation 19-24 makes pixel-wise classification and provides more explainable results for the subsequent decision making process.

METHOD
In the black-box UDA segmentation task, we have a network f s trained with the labeled source domain sample {x s , y s }, where f s is fixed and accessed only through a nontransparent API at the adaptation stage. At the adaptation stage, we only have access to the black-box f s and the unlabeled target domain sample {x t } to train a target domain network f t to achieve a good segmentation performance in the target domain. In this work, we propose a practical solution to the black-box UDA for segmentation with a noise-aware knowledge distillation scheme using pseudo label with exponential mixup decay (EMD). The framework is shown in Fig. 1.
Following the knowledge distillation, 4 the well-trained source model can act as a teacher to provide its pixelwise softmax histogram prediction of each image. The target domain model is trained to imitate the source model. The consistency of their prediction can be enforced with the Kullback-Leibler (KL) divergence between their pixel-wise softmax histogram distributions. However, due to the domain shift, the prediction of f s can be noisy. Accordingly, we resort to a self-training scheme 1 to construct the pseudo label for target domain training. Specifically, to achieve the gradual translation to the target domain, we mix up the source and target domain predictions, i.e., f s (x t ) and f t (x t ), and adjust their ratio for the pseudo label y t with EMD: where n indexes the pixel, and f s (x t ) n and f t (x t ) n are the histogram distributions of the softmax output of the n-th pixel of the predictions f s (x t ) and f t (x t ), respectively. λ = λ 0 exp(−I) is a target adaptation momentum parameter with an exponential decay w.r.t. the iteration I. λ 0 is the initial weight of f s (x t ), which is empirically set to 1. The loss knowledge distillation with the EMD pseudo label can be formulated as: where H 0 and W 0 are the height and width of the image. Therefore, the weight of λ can be smoothly decreased along with the training, and f t gradually represents the target data. In addition, an unsupervised training scheme, e.g., entropy minimization, 25 can be added on top of our framework. For implementation, the entropy for pixel segmentation can be formulated as an averaged entropy of the pixel-wise softmax prediction, given by Minimizing L Ent leads to the output f t (x t ) n close to a one-hot distribution, i.e., confident prediction.
In summary, our training objective can be formulated as where α is used to balance between the knowledge distillation with the EMD pseudo label and entropy minimization. We note that a trivial solution to entropy minimization is that all unlabeled target data could have the same one-hot encoding. 25 Thus, in order to stabilize the training, we linearly change the hyper-parameter α from 5 to 0.

RESULTS
We evaluated our approach on the BraTS2018 database, comprising a total of 210 high-grade glioma (HGG) subjects and a total of 75 low-grade glioma (LGG) subjects. 26 Each subject has T1-weighted (T1), T1-contrast enhanced (T1ce), T2-weighted (T2), and T2 Fluid Attenuated Inversion Recovery (FLAIR) Magnetic Resonance Imaging (MRI) volumes with voxel-wise labels for the enhancing tumor (EnhT), the peritumoral edema (ED), and the necrotic and non-enhancing tumor core (CoreT). We denote the sum of EnhT, ED, and CoreT as the whole tumor. More information about the database can be found in. 26 Training was performed on four NVIDIA TITAN Xp GPUs with the PyTorch deep learning toolbox, 27 which took about 5 hours.
Following the previous white-box source free UDA 3 and UDA with source data, 28 we used HGG subjects as the source domain and the LGG subjects as the target domain, which have different size and position distributions. 28 We trained f s using the prior work, 3 and did not have access to its network parameters at the adaptation stage. For simplicity, we used the same 2D U-Net backbone for f s and f t as in. 28 Specifically, we used U-Net 29 as our segmentation network with 15 layers, batch normalization, and dropout. The network was trained using Adam as an optimizer with β 1 = 0.9 and β 2 = 0.99.
The slices of four modalities were concatenated as a 4-channel input with the spatial size of 128×128. The training of adaptation used the LGG training set. The evaluation was implemented in the LGG testing set. For Figure 2. Comparison with the other UDA methods, and an ablation study of entropy minimization for HGG to LGG UDA. We use white, dark gray, and gray to indicate the CoreT, EnhT and ED, respectively. OSUDA 3 with the white-box source model for UDA training is regarded as an "upper bound." evaluation, we adopted the two metrics, including Dice similarity coefficient (DSC) and Hausdorff distance (HD) metrics. 30 We used BBUDA to denote our black-box UDA framework, and BBUDA-Ent indicates the ablation study without entropy minimization. An illustration of the segmentation results is shown in Fig. 2. We can see that the predictions of our proposed BBUDA are better than the no adaptation model. The better performance of BBUDA over BBUDA-Ent demonstrates the effectiveness of our entropy minimization.
The quantitative evaluation results are shown in Table 1. Our proposed BBUDA achieved the state-of-theart performance for the black-box source-free UDA segmentation, approaching performance of the white-box OSUDA 3, 16 with the source model parameters, which can be considered an "upper-bound." We note that the label ratio consistency assumption in CRUDA 16 does not hold for this HGG to LGG transfer task, which thus led to an inferior performance.
In addition to the HGG to LGG setting, we also proposed to adapt from LGG to HGG. Similar to the HGG to LGG task, there was also the domain gap w.r.t. tumor types and the label proportion of each class. The results are shown in Table 2. Our proposed BBUDA achieved superior segmentation performance consistently.

CONCLUSION
In this work, we tackled the problem of black-box UDA for segmentation under a realistic and meaningful scenario, and proposed a practical and efficient knowledge distillation scheme with EMD pseudo label. Specifically, we were able to smoothly transfer the segmentation in the source domain to the target domain with EMD to construct the pseudo label. In addition, unsupervised entropy minimization was added on top of our model to further boost the performance. Experimental results, carried out with the HGG to LGG adaptation task, showed that our proposed BBUDA outperformed the source model, by a large margin, and importantly, the DSC and HD metrics were comparable to the white-box UDA approaches. In this work, we only investigated the adaptation between HGG and LGG data, while the model can be general for any segmentation UDA tasks. In addition, more advanced knowledge distillation and unsupervised learning methods can be added on to further boost the performance, which is subject to our future work.


s2fieldsofstudy:
Computer Science
Physics

text:
Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics

Measuring the similarity of images is a fundamental problem to computer vision for which no universal solution exists. While simple metrics such as the pixel-wise L2-norm have been shown to have significant flaws, they remain popular. One group of recent state-of-the-art metrics that mitigates some of those flaws are Deep Perceptual Similarity (DPS) metrics, where the similarity is evaluated as the distance in the deep features of neural networks. However, DPS metrics themselves have been less thoroughly examined for their benefits and, especially, their flaws. This work investigates the most common DPS metric, where deep features are compared by spatial position, along with metrics comparing the averaged and sorted deep features. The metrics are analyzed in-depth to understand the strengths and weaknesses of the metrics by using images designed specifically to challenge them. This work contributes with new insights into the flaws of DPS, and further suggests improvements to the metrics. An implementation of this work is available online: https://github.com/guspih/deep_perceptual_similarity_analysis/

I. INTRODUCTION
Similarity metrics are a fundamental part of many machine learning processes. Every time two or more objects are compared, a similarity metric is used. In computer vision, widely used metrics, such as the pixel-wise L2-norm, have been carefully studied and their benefits and flaws are wellknown which lets users make an informed decision when using them.
Many improvements to pixel-wise metrics have been, with a common goal being to mimic human perception with a so-called perceptual similarity. One popular perceptual similarity metric is the Structural Similarity Index Measure [1]. A more recent approach is to utilize deep features learned by machine learning models for measuring perceptual similarity. This practice, called Deep Perceptual Similarity (DPS) measures the similarity of two images by comparing their respective activations in the deep layers of Convolutional Neural Networks (CNNs), instead of using the pixel values directly.
DPS metrics have outperformed previous models on perceptual similarity [2]. Additionally, such metrics have been used as part of the loss function for training models, which have achieved impressive results on a host of tasks. These tasks include, image generation [3], style transfer and superresolution [4], object detection [5], and image segmentation [6].
While there are clear benefits of DPS, its flaws are not as well studied. While it has been shown that deep perceptual similarity 1 https://github.com/guspih/deep_perceptual_similarity_analysis/ is vulnerable to adversarial examples, this is expected from any method depending on deep networks and existing methods for protecting from adversarial attacks such as ensembles may be utilized [7]. Additionally, adversarial examples are quite complex compared to the known flaws of other metrics. For example, pixel-wise metrics would consider a black-and-white image to be as dissimilar as possible from its inverted version.
This work aims to analyze if and how DPS can successfully handle the flaws of the pixel-wise L2-norm, and investigate if there are any similar unexplored flaws of DPS and how those may be mitigated. Additionally, several different DPS metrics are analyzed for flaws and then evaluated on the BAPPS dataset [2], to check if those flaws translate into performance on an actual dataset.
The investigation of DPS is performed by creating image pairs that are similar to each other compared to some reference images and checking in which cases the DPS metrics succeed or fail in identifying the image pairs as more similar than the reference. The feature maps of the CNNs used for calculating similarity are analyzed to gain insight as to what underlies the successes and failures.

II. RELATED WORK
Commonly used image similarity metrics are pixel-wise metrics where each pixel of one image is compared directly against the corresponding pixel of the other. These metrics have long been known to be poor similarity metrics as they disregard high-level image structures [8,9,10]. Instead many different perceptual similarity metrics have been proposed including Dynamic Partial Function [11], the Structural Similarity Index Measure [1], and Structural Texture Similarity [12]. Despite known flaws and suitable alternatives, per-pixel metrics have consistently been used for image comparison within computer vision in general, and to calculate the loss for machine learning models specifically.
One powerful attribute of deep learning is that the deep features learned by the networks typically contain information useful for other tasks than the one the network was trained for. This attribute was used to great effect with the introduction of neural style transfer, where the content and style of images were compared using different sets of deep features within a neural network [13]. This practice of training models to minimize the difference between the activations of a deep network in order arXiv:2207.02512v1 [cs.CV] 6 Jul 2022 to get visually similar images is known as deep perceptual loss.
Deep perceptual loss has since its introduction been successfully applied to a large number of computer vision tasks such as improving the performance of variational autoencoders [14,15,16], Generative Adversarial Networks [3], Super-Resolution [17,18], and style transfer [4]. The method has been proven effective at the task of perceptual similarity where it significantly outperformed previous methods [2]. This method of calculating perceptual similarity using the deep features of neural networks is referred to as deep perceptual similarity (DPS).
One potential problem with DPS is that it relies on deep neural networks, which are known to be vulnerable to adversarial examples. Adversarial examples are almost imperceptible perturbations to images or other input data that induce significant changes or errors to the prediction model [19]. While no perfect protection from adversarial examples is currently known, there is a wide array of defenses that can be used, including using ensembles [7]. Additionally, outside of malicious attacks, this is rarely a problem.
Another paradigm for creating similarity metrics is to optimize a machine learning model for the task [20]. This has been applied to DPS with the LPIPS method, though it notably only performed marginally better than using methods that had only been pretrained [2]. Like with many other machine learning methods the results can be improved somewhat with the use of ensemble methods, though still comparable to pretrained models [7].
Where this work analyzes DPS through deep analysis of cases where it fails, another recent work investigates how different network architectures and pretraining procedures affect performance [21]. That work found, among other things, that better pretraining performance on ImageNet [22], does not necessarily lead to better perceptual similarity, It additionally showed that a good pretrained model can outperform models trained specifically for the similarity task.
As DPS metrics inherently rely on the deep activations of neural networks, most commonly CNNs, analyzing these activations is inherently interesting. Many methods for such analysis exist and one of the most common is to visualize the feature maps of the CNNs [23], which is utilized in this work.

III. DEEP PERCEPTUAL SIMILARITY
Most uses of deep perceptual similarity and deep perceptual loss have directly compared the corresponding activations of the two images. This method, referred to as spatial DPS, is formalized as the distance measure between x and x 0 in Eq. 1, where f is a norm such as L1 or L2 and p is a convolutional feature extractor with extraction layers l ∈ L each with C l channels with height H l , and width W l .
This work evaluates two additional methods of calculating deep perceptual similarity besides the spatial method. These two are the mean method tested in [21] and a sort method which is introduced in this work. The two methods are formalized in Eq. 2 Eq. 3 where x and x ↓ are the average and descending reordering of x respectively.
Both of these methods ignore the spatial positions of the features. The mean method compares the average of the features in each channel and the sort method pairs the features of each channel with one another in such a way as to minimize the norm. In the sort method the norm is minimized for any convex function f , compared to any other ordering of the features. This follows from In the case of infinitely large input images and translationinvariant CNNs the two presented methods are translationinvariant as no matter how much the image is translated the same features will appear. In the case with bounded images, as long as regions with the strongest feature activations aren't shifted off the image or too close to the boundaries, this should still likely result in a metric that is robust to translations. Even though many CNNs aren't strictly translation-invariant, in general translations have very little effect on the methods. The reasoning behind comparing average and sorted channels is that a strong activation in one channel often represent different concepts than a similar activation in another.
A problem with the mean and sort methods on their own is that humans would likely say that a lower translation is more similar to the original than a greater one. As such complete translation-invariance is not desirable. Thus, this work also investigates metrics that uses the sum of the spatial method with one of the two non-spatial methods.

A. Experimental Setup
DPS relies on neural networks which deep features contain useful information for image comparison. While networks can be trained specifically for the task, the most common use of DPS and deep perceptual loss is pretrained networks.
This work uses mostly the same feature extraction and comparison setup as [2]. The methods are analyzed and evaluated with the L2-norm as the comparison function (f ) using three models (p) pretrained on the ImageNet dataset [22]. The architectures for the three models are SqueezeNet [25], AlexNet [26,27], and VGG-16 [28]. The deep features are extracted from the same multiple layers for each network as in [2]. The features extracted in the original work were channelwise unit-normalized, and this work analyzes and evaluates both using and ignoring this practice. However, for brevity the analysis in Section IV concerns only the case without unit-normalization and the use of unit-normalization is later discussed in Section VII.

IV. QUALITATIVE ANALYSIS OF DPS ON DISTORTIONS
This work carries out a qualitative analysis of deep perceptual similarity metrics over images specifically designed to test for its strengths and potential flaws. The analysis is carried out by distorting images in ways for which DPS is previously known to work well or speculated to perform poorly. The similarity of the distorted image with the original is then compared to the similarities of a set of reference images and the original, where the reference images are intended to be notably less similar than the distortion. The feature maps at various layers of the DPS networks are then analyzed for each case to gain a deeper understanding of why the metric performed the way it did in this case. Such insight was then used to create further image pairs to test against. Finally, for one category of images, specific reference images were created for each image pair. These reference images, like the others, were created to be perceived by humans as less similar than the distorted versions but intended to fool some DPS metrics.
The images used in the tests are 96 × 96 pixels and have been designed and distorted by hand. The distortion tested are divided into four categories; color inversion, translation, rotation, and color stain. Seven reference images were created; mono-colored images of black, white, gray, red, green, and blue, as well as one with randomly colored pixels. One image pair each from the inversion, translation, and rotation categories is shown in Fig. 1

A. Black-and-White Color Inversion
Color inversion of black-and-white images is typically used as an example of when pixel-wise metrics break down. This is because each pixel in the inverted image is as different from each other as they could be, which means every other possible image would be regarded as more similar, which is obviously not the case for perceptual similarity. Despite this being used as an example of why to abandon pixel-wise metrics in favor of DPS metrics, there has been little investigation of how well DPS performs in these scenarios. For these reasons the first set of images created for analyzing DPS were simple black-and-white patterns that were distorted by inverting the colors.
While pixel-wise metrics fail by definition on this category of images, all tested DPS metrics correctly identify each image pair as more similar than any of the reference images. Analysis of the feature maps reveals that many channels are activated by contrasts or higher-level structures like lines or shapes. These activations are often completely agnostic to inversion and identify the structures regardless of color. This makes the black-and-white inversion pairs almost exactly the same for many channels in the feature space, which leads to the good performance of DPS on color inversion.

B. Translation and Rotation
It is also clear from the feature maps that all activations are strongly spatially correlated to where those features appear in the input image, which can be seen in Fig. 3. This is obvious as CNN architectures in general are built around each activation depending only on a small region of the input or previous layer. While, in theory, activations in the later layers depend on information aggregated from a large swath of the image, in practice, strong activations in the feature maps at any layer are correlated with features in the spatially corresponding region of the input image. This has been previously suggested as a potential flaw of spatial DPS [21].
To investigate whether this would have a significant impact on spatial DPS and whether other DPS metrics could handle these cases, the categories of translation and rotation have been tested. The translation images have a region containing much structure in otherwise plain images which have been distorted by translating that region. The rotation images are simply images that have been distorted by rotation in steps of 22.5 up to 90 degrees, as well as one rotated 180 degrees. The purpose of the incremental steps was to see if and further how sensitive DPS is to rotation.
Both the pixel-wise metric and spatial DPS fail to identify any translated image as more similar than the reference images, while the other DPS metrics succeed in each case. For rotation, both pixel-wise and spatial DPS metrics fail on about the same amount of cases, slightly less than half, while the other DPS metrics almost succeed on each image pair.
This clearly shows that the spatial DPS metric on its own is not suitable for these types of scenarios, while translationinvariant DPS metrics can handle them very well. It is also interesting that the translation-invariant DPS metrics handle rotation so well since it is well known that early channels in CNNs often learn to identify specific orientations in lines and other structures. Likely, the later layers of CNNs combine orientation-specific features into higher-level orientationindependent ones.

C. Color Stain
Another revelation from feature map analysis is that many channels tend to activate strongly from specific colors, textures, or random noisy structures. This might be challenging for nonspatial methods as ignoring the spatial position of activations might lead to confusing noise for interesting structures. For example, in the case of mean DPS, an image with a small interesting region might be seen as dissimilar from the same with added stains since the average activation in the noisy one will be larger.
To test for this the color stain category is used. The image pair for the color stain category consists of a plain image with a structurally interesting region, and a distorted version with a similar or same interesting region but the plain color is changed, and added noisy features for some images. The color stain category does not use the same reference images as the other categories, and instead, each image pair has a specific reference image designed to be less perceptually similar. These reference images have the same plain color without stains as the non-distorted image, but their interesting region is significantly different compared to the distorted version. Examples of image pairs and their specific reference image are shown in Fig. 2.
For the color stain category, the pixel-wise metric again fails for each image pair. Notably, both the mean and sum of spatial and mean DPS fails almost all image pairs. The remaining DPS metrics tested perform well, with spatial DPS being the best.
One specific image in this category was a white image with a red, green, and blue irregular circle in one corner. The distorted image retained the circles but the plain white background was colored a darker shade of green with random yellow stains. By observing the feature maps of these images it is clear that the color change and stains add significant activations to the otherwise sparse feature maps, especially in later layers. This is shown in Fig. 3, where the image and its distortion are displayed together with feature maps from the second and fourth SqueezeNet ReLU layer.

V. EVALUATION
In order to investigate how the insights gained through the qualitative analysis translate to performance on a perceptual similarity dataset, the DPS metrics are evaluated on the BAPPS dataset, using the same procedure as in the original work [2]. However, the evaluation is only carried out for the pretrained networks SqueezeNet, AlexNet, and VGG-16 without additional fitting to the training data. This is equivalent to what the original work refers to as "Net (Supervised)", with the addition of testing non-spatial DPS metrics. This evaluation follows the original work which means f is the L2 norm and the features extracted from p have been channel-wise unit-normalized.
2AFC consists of image patches and two distorted versions of each patch, as well as human annotations as to which distorted patch is most similar to the original. The aim of 2AFC is to train and evaluate models for perceptual similarity judgment by evaluating if those models give higher similarity to the distortion that most human annotators agreed was more similar. In addition to evaluating on the complete 2AFC part, results are also given for a number of subdivisions defined by the type of distortions that are applied: (1) Traditional augmentation methods, outputs from (2) CNN-based autoencoders, (3) superresolution, (4) frame interpolation, (5) video deblurring, and (6) colorization.
JND consists of an image patch as well as a barely distorted version along with human annotations of whether the two patches are the same. The human annotators were shown the two images only briefly and were also shown pairs of the same and very different images. The aim of JND is to test models for perceptual similarity by evaluating if those models give a higher similarity to those samples that human annotators had difficulty telling apart.

VI. RESULTS
An aggregation of the outcome of the tests described in Section IV is shown in Table I. The performance is presented as the number of images, where the metric did not find any of the reference images to be more similar than the distorted version.
The results of the evaluation on the BAPPS dataset are shown in Table II for each subdivision of the 2AFC part, for the entire 2AFC part given by the average over the subdivisions, and for the JND set. The results for the evaluated metrics are presented along with the LPIPS metrics from [2] and human performance for reference.

VII. DISCUSSION
The purpose of this work has been to evaluate if and how DPS metrics can handle the typical cases where pixel-wise metrics fail, and to investigate whether similar flaws exist in current DPS implementations. All tested DPS metrics handle the color inversion tests for which pixel-wise metrics break down, and additionally the clear preference for contrasts and structures in feature maps indicates that DPS metrics are well-suited to handle other similar color-changing operations. By far most common form of DPS metrics used is spatial DPS, which seems to perform as poorly as the pixel-wise metric on the rotation and translation test cases. While the non-spatial DPS metrics perform well on these weaknesses, they do not perform as well as spatial metrics on the color stain category of tests. This is especially true for mean DPS which failed most of the color stain tests. The spatial and non-spatial combined metrics perform similar to spatial DPS, indicating that perhaps combining metrics using unweighted summation gives a preference for spatial DPS. Though the results of the combined metrics improved somewhat for rotation, indicating that there are at least some benefits to this strategy.
Analyzing the BAPPS scores for the different DPS metrics shows that flaws of spatial DPS also affect performance on a perceptual similarity dataset. While spatial DPS, in general, performs worse than the other DPS metrics, notably, this is especially true for the traditional augmentations subdivision. Traditional augmentations include operations such as rotation, translation, and skewing which indicates that the weaker performance of spatial DPS is due to the flaws identified in this work.
Another notable result is that mean DPS, on average, performs best on BAPPS, even though it was vulnerable to the color stain category of distortions to a much larger degree than sort DPS. However, both mean and sort DPS metrics perform similarly and are both better choices than spatial DPS. It is possible that color stain and related distortions are not so common to be a problem in a real-world scenario, or that the BAPPS dataset does not include such cases. Additionally, the image pairs used for analysis in this work have often been very simple and plain compared to the images typically included in datasets.

A. The effects of unit-normalization
As mentioned in Subsection III-A, the qualitative analysis described in Section IV was also performed with channel-wise unit-normalization of the extracted features. This had three notable effects. First, the success rate in the rotate category rose for all DPS metrics, especially for the metrics that use spatial DPS. In fact, those metrics became almost competitive with mean and sort DPS. Second, while spatial still fails on each image pair in the translate category, the combined metrics are somewhat improved. Likely due to normalizing making the spatial distances lower which gives more weight to the non-spatial distances. Finally, using normalization made each DPS metric perform poorly in the color stain category.
When evaluating with the BAPPS dataset unit-normalization has only a small positive effect on performance. Likely translation and rotation and similar augmentation are more common in that dataset than augmentations similar to the color stain procedures.

VIII. FUTURE WORK
From the results and analysis presented in this work there are some notable directions of research to explore.
Both this and a prior work [21] has shown that spatial DPS does not perform as well as on perceptual similarity tasks as other implementations of DPS. One future possibility is to investigate if this translates to related field such as deep perceptual loss and content-based image retrieval. If it does, simply changing the way perceptual loss is calculated could improve the results on many different tasks.
While most DPS metrics outperform previous perceptual similarity metrics, the discrepancy in performance of DPS metrics indicates that exploring how to calculate DPS metrics is an open problem. For example, a DPS metric that make use of both spatial and non-spatial comparisons could perhaps gain the benefit of both. Additionally, the upsides and downsides of unit-normalization remain inconclusive.