Geometrically Guided Integrated Gradients

Interpretability methods for deep neural networks mainly focus on the sensitivity of the class score with respect to the original or perturbed input, usually measured using actual or modified gradients. Some methods also use a model-agnostic approach to understanding the rationale behind every prediction. In this paper, we argue and demonstrate that local geometry of the model parameter space relative to the input can also be beneficial for improved post-hoc explanations. To achieve this goal, we introduce an interpretability method called"geometrically-guided integrated gradients"that builds on top of the gradient calculation along a linear path as traditionally used in integrated gradient methods. However, instead of integrating gradient information, our method explores the model's dynamic behavior from multiple scaled versions of the input and captures the best possible attribution for each input. We demonstrate through extensive experiments that the proposed approach outperforms vanilla and integrated gradients in subjective and quantitative assessment. We also propose a"model perturbation"sanity check to complement the traditionally used"model randomization"test.

Introduction
The past decade has seen a wealth of new advancements in deep learning (DL), improving performance in a wide array of possible problems in many areas, especially in classification tasks from computer vision and natural language processing [Chai et al., 2021, Young et al., 2018. However, this improved performance comes with a cost, the models are not easily explained or described with simple observations. This lack of intelligibility hinders wide-scale use of these models in safetycritical domains such as healthcare, education, the justice system, and many others. This need for decipherability has given way to parallel advancements in a subfield of machine learning known as interpretability. Interpretability has grown as quickly as the DL field itself, providing many new state-of-the-art techniques [Baehrens et al., 2010, Simonyan et al., 2013, Shrikumar et al., 2017, Sundararajan et al., 2017, Selvaraju et al., 2017, Bach et al., 2015, Montavon et al., 2017, Kapishnikov et al., 2019 Interpretability research generally tackles these problems by either describing the model's decisionmaking processes or by generating post-hoc explanations, either for the model as a whole, or for each sample. There are many post-hoc explanation methods from research spanning the last 2 decades. The performance and properties of these methods vary widely across different architectures and domains. In recent years, beginning with integrated gradients (IG) [Sundararajan et al., 2017], some of these methods [Kapishnikov et al., 2019, Xu et al., 2020 have satisfied two important properties: sensitivity and implementation invariance.
These post-hoc analysis methods, specifically the ones that leverage model gradients to explain each sample, are known as saliency techniques. Vanilla gradients (GRAD) [Baehrens et al., 2010, Simonyan et al., 2013 and IG, two of the most popular methods, can be noisy [Montavon et al., 2017, Samek et al., 2016, Smilkov et al., 2017, Sturmfels et al., 2020. We show an example of this noise in Figure 1. GRAD specifically violates both implementation invariance and sensitivity. IG also has specific noise relating to the averaging or integrating over the interpolation path. Other sources include the large curvature of the network's decision function [Dombrowski et al., 2019], numerical approximation of integration [Kapishnikov et al., 2021], and baseline choices [Xu et al., 2020, Sturmfels et al., 2020. In order to both conform to the two aforementioned properties and reduce noise, we suggest that the gradients can be improved by leveraging more aspects of the model space than the exact sample loss or a linear interpolation of the sample as is found in IG. Specifically, geometric properties of the loss landscape, which have previously been utilized to strengthen model performance and robustness , Gotmare et al., 2018, Entezari et al., 2021.
In this paper, we propose Geometrically Guided Integrated Gradients (GGIG), an algorithm that builds on top of traditional IG to reduce noise. We expand IG by ascending through the loss space by maximizing the class-specific logit, much the way class activation maximization [Couteaux et al., 2019] works. This ascension, or maximization of the class logit, allows GGIG to find the gradients that are most discriminative, which we suggest is a valuable property for any interpretability mechanism. After describing the method, we empirically show that GGIG improves the quality and robustness of the saliency maps. Figure 2 describes the general pipeline of the work.

Input
Integrated Gradients Gradients GGIG Figure 1: Comparison of gradients, integrated gradients and GGIG based on top 5% pixel attributions for the "gazelle" prediction. In GGIG explanation, the details of the animal are clearly noticeable and discarded the undesirable regions from the explanation.
Our main contributions are as follows: • We propose an interpretability method, called Geometrically Guided Integrated Gradients, that starts with linear path as used in IG and finds the path that enhances the class activation for the underlying prediction.
• We also propose a model perturbation sanity check, also called σ-perturbation that we think all explanation methods should satisfy.
• We show that the proposed method offers better saliency maps for different datasets and architectures when assessed through visual inspection and quantitative metrics.
GradCAM [Selvaraju et al., 2017] identifies the focal regions pretty well, and maps are highly predictive. However, the saliency maps are blobby [Kapishnikov et al., 2019] and usually expand around the actual objects. While IG has its own specific problems, several recent studies have refined IG attributions because this method has many desirable properties. Kapishnikov et al. [2019] proposed a region-based attribution method, called XRAI, that mainly refines the IG attribution based on attribution density. However, XRAI requires a way to cluster the input features, which may not be available for different data modalities. Kapishnikov et al. [2021] proposed another method, called GIG, which provides an adaptive path method based on input, baseline, and the model. This method starts at the baseline and selects only those pixels with the lowest partial derivatives to take closer to the next interpolation point, thus avoiding the gradient accumulation from saturation regions. In other words, it constitutes the path based on dynamic projections of the linear interpolation path.
To reduce the inherent noise in the saliency maps, we may utilize some useful loss landscape properties as observed in several studies , Gotmare et al., 2018, Entezari et al., 2021 to design reliable interpretability methods.

Explanation Generation Explanation Evaluation
Class Prediction Model GGIG Estimator Figure 2: Overall pipeline of the work. First, we generate explanations for each prediction. Secondly, for quantitative evaluation, we start with a complete defocused image and combine with only salient pixels (thresholded at x%) to create a test image. Finally, we feed the model with the test image and measure both accuracy and relative softmax scores.

Geometrically Guided Integrated Gradients (GGIG)
We propose a method called GGIG, which incorporates the idea of path methods [Sundararajan et al., 2017] and enhances the quality of the attribution by analyzing the local loss behavior. Figure 3 shows the schematic diagram illustrating the functional mechanism of GGIG.
Like IG, GGIG starts from the baseline x and constitutes a linearly interpolated path. However, instead of accumulating gradients along the path, it updates the path gradients in the direction where the model maximizes class activations. GGIG thus maximizes class activation for each of the linearly interpolated points. The procedural steps for GGIG are shown in Algorithm 1. We hypothesize that the prediction curve in the vicinity of x holds important information about the interaction between model f and input x.
Let F : R d → R be defined as the mapping from the input space to the class-specific logit and b 0 , . . . , b n be a set of n linearly interpolated points for each sample between the baseline, b 0 and the exact sample for k ← 0 to n do 4: return e e is GGIG explanation 12: end procedure where j is the incremental ascent over the sample and lr is the learning rate. This ascending mechanism allows the model to take the direction where logit values are enhanced. All gradients over the ascension, ∇F (b j i ), are collected, and the max ∇F (b j i )[p] (for each pixel p), where i = 0, 1, 2, . . . , n and j = 0, 1, 2, . . . , m, is selected as the final attribution for the pixel p.
Logit for the prediction : input : baseline Figure 3: The basic idea of "Geometrically Guided Integrated Gradients." It takes a baseline x , creates a linearly spaced path to the actual input x. From each interpolation point, it follows a trajectory in the loss landscape that maximizes class activation.
IG [Sundararajan et al., 2017] has many desirbale properties and we have used IG as the baseline to compare the proposed method. We used slightly different formulation of IG. Generally, IG uses interpolation technique to integrate importance at different discrete intervals between uninformative baseline, sayx and the input x, to give an integrated estimate of feature importance. IG based feature importance is computed as: The ultimate estimate e depends on the value of k (number of intervals) and the choice of a suitable uninformative baselinex. The traditional integrated gradients scale raw attributions (operand on the right of the multiplication operator) by x − x (operand on the left).
Element-wise multiplication is misleading: Adebayo et al. [2018b] observed that element-wise multiplication could be misleading. This misleading happens mainly because the input dominates the product even with drastic changes in gradient vectors. So, the interpretability methods with this element-wise multiplication component in their formulation can provide input-dominant explanations that may deceive human understanding.
Furthermore, Ancona et al. [2019] suggested that this point-wise multiplication was initially justified to sharpen the gradient explanations; however, it is better justified when a measure of salience is a priority over mere sensitivity. In this paper, we were more interested in the sensitivity of features rather than their marginal salience to the target score. For all of these valid reasons, we did not multiply the integrated gradient with x − x to avoid input dominance from the explanations. Moreover, we did not consider other gradient-based methods like grad input for the same reason.
The baseline can be problematic for correct attributions [Kapishnikov et al., 2019, Sturmfels et al., 2020. Kapishnikov et al. [2019] addressed the issue of baseline by using both black (0, 0, 0) and white (1, 1, 1) baselines. While baseline may be an issue for traditional IG formulation, avoiding direct or modified (as in IG) element-wise multiplication resolves the baseline issue from this work.

Model Perturbation Sanity Check
In this section, we propose a model perturbation sanity check for attribution methods. We refer to this perturbation as σ-perturbation. This perturbation seeks two important properties: We claim that all attribution methods should satisfy this sanity check to ensure their meaningfulness and sensitivity to the model parameters. We applied this sanity check to vanilla gradients, integrated gradients and GGIG. Interestingly, all of the methods satisfied the σ-perturbation sanity check.
We report σ-perturbation sanity check results in Figure 4.
Explanations generated with model perturbations Relative Softmax Score Figure 4: Left: Comparison of saliency-focused images thresholded at 10% while we perturb the model. Right: Relative softmax score during model perturbation. As expected, the similarity with the original explanations breaks and softmax score drops with gradual increase in perturbation level.

Experiments
Training on MNIST Dataset For MNIST [LeCun, 1998], the model architecture was a CNN that consisted of two convolutional layers with (32, 64) filters of sizes (5, 5). Each convolutional layer is followed by a 2 × 2 max pooling layer and a ReLU activation. We fed the final convolution output to a fully connected network with 1024 input and 10 output units (softmax). We optimized the model using stochastic gradient descent (SGD) with a learning rate of 0.0004 and momentum of 0.9. The model was trained for 400 iterations with a mini-batch size of 64 and finally achieved an accuracy of 99.2%.
Post hoc explanation experiments For GGIG, we used a learning rate of 0.0001 for gradient ascent from each linear interpolation point between input x and baseline x = 0. For MNIST, we iterated the gradient ascent for 200 steps and noted the maximum sensitivity along the gradient ascent trajectory for each input. We display the saliency maps obtained on the MNIST dataset in Figure 11.
Quantitative Evaluation on MNIST Dataset Visual inspection of explanation methods can be unreliable as it is possible to create adversarial samples [Goodfellow et al., 2014, Szegedy et al., 2013] that can fool the human eye, totally changing the model predictions. We perform different similarity measures between the maps and the input to understand the quality of the proposed methods on MNIST, namely, Spearman Rank Correlation, Weighted Jaccard Similarity, Structural Similarity, and Normalized (Reverse) Mean Square Error. For quantitative evaluation, both data and maps were rescaled in the range [0, 1]. We assumed that the amount of information a method can capture about the structure and distribution of the input in the saliency maps directly determines its quality as an explanation method. We report the quantitative evaluation in Table 1. Figure 11 in Appendix shows sample maps and the detailed results of quantitative evaluation. As we can observe, GGIG outperforms other methods. Comparatively, GRAD and IG retain little information about the numerical and structural association to the input. Experiments on ImageNet: We also evaluated the proposed methods using a subset of images from the ImageNet dataset [Krizhevsky et al., 2012] and different pretrained models, namely ResNet-101 [He et al., 2016], and Inception V3 [Szegedy et al., 2016]. Though we found meaningful maps in every case, maps still vary in quality possibly due to their architectural differences. Figure 5 and Figure 6 show some maps (more results are in Appendix) produced using different saliency methods and pretrained models. Figure 5: Sample saliency maps for ImageNet Validation set and ResNet101 model generated by gradients, integrated gradients, geometrically guided integrated gradients, and Edge Detector.

Model Randomization Test:
We also performed a Model Randomization test [Adebayo et al., 2018b] (Figure 7) to verify the sensitivity of the methods to the model parameters. To this end, we randomly reinitialized the weights and generated post-hoc explanations using the randomized model. The proposed method GGIG is as sensitive as GRAD and IG, suggesting that our method is highly sensitive to model parameters.
Saliency methods are not edge detectors: Adebayo et al. [2018b] observed that many saliency methods, including vanilla gradients and integrated gradients can appear like edge-detectors for 1-Layer Sum-Pool Conv Model. This assumption may only hold for shallow models. To analyze the edge-detector like behavior for deep models, we conducted an experiment where we replaced the original background of the images with a sharp-changing image. In particular, we assigned ImageNet samples a very different fixed background (a black and white checkerboard) and generated post-hoc explanations using GRAD, IG, and GGIG. We show the resulting explanations in Figure 8. It is obvious from the resulting maps that the model used concepts, not merely edges from the training objects. Moreover, as expected, all the saliency methods ignored the background and only attributed the object's parts for prediction explanations. It is apparent that GGIG method retained learned concepts more accurately during post-hoc explanations. We think that for well-trained deep models, saliency methods no-longer function like an edge-detectors. Figure 6: Comparative saliency maps for Inception V3 model and ImageNet dataset generated by GRAD, IG, GGIG, and Edge Detector. The maps obtained using GGIG are more discriminative and more clearly reveal the underlying structure of the class-associated objects.

Evaluation of Attribution Quality
Visual Analysis: We show few sample explanations thresholded at 10% in Figure 9. As we can observe, GGIG provides the least noisy explanations compared to other methods. As expected, the edge detector does not pay attention to discriminative regions. Rather, it captures only the sharp changes throughout the image. While GRAD and IG attribute lots of redundant or unexpected parts of the image, GGIG directly points to the most discriminative pixels of the image. Moreover, GGIG mostly avoids sharp changes in the image (see the explanation for the "leopard"), while GRAD and IG are highly susceptible to the edges. Figure 9: Comparison of saliency-focused images thresholded at 10%. GGIG, compared to other methods, is more convincing from the human standpoint because it directly points to the discriminative parts of the image while ignoring redundant and unexpected regions.
Quantitative Evaluation: It is challenging to evaluate an interpretability method because of the lack of ground truth saliency or consensus metrics for proper evaluation. Several studies proposed different measures, such as Remove And Retrain (ROAR) [Hooker et al., 2019], Retain And Retrain (RAR) [Rahman et al., 2022], Accuracy Information Curves, Softmax Information Curves [Kapishnikov et al., 2019] to assess the quality of explanations.
The ROAR approach modifies the dataset by removing the features that received top attribution values from each sample. In practice, if the training data have sufficient (redundant) discriminative features [Sturmfels et al., 2020], even after removing a significant number of features, the performance of the retrained model does not drop noticeably. In that scenario, ROAR fails to evaluate the feature relevance correctly. Retain and Retrain (RAR) method resolves the problem by retaining only the critical features instead of removing them. However, both ROAR and RAR methods are time-consuming as they require full retraining of the model. Dabkowski and Gal [2017] proposed a metric, called Smallest Sufficient Regions (SSR), based on the notion of the smallest sufficient region capable of correct prediction. However, this metric is not suitable if the model is susceptible to the scale and aspect ratio of the object. Moreover, as this metric depends on rectangular cropping and reports results as a function of the cropped area, this approach highly penalizes if the saliency map is coherently sparse [Kapishnikov et al., 2019]. Because, in that case, it may span a larger area of the image than the map, which is locally dense, even with the same number of pixels. SSR also imposes a severe challenge because masking creates a sharp boundary between the masked and salient region, causing an out-of-distribution problem for the model. Kapishnikov et al. [2019] proposed two metrics, called Accuracy Information Curve and Softmaxe Information Curve, collectively called Performance Information Curve (PIC), to evaluate the saliency maps with the minimal out-of-distribution setting. We used SIC and AIC to evaluate our method and adopted a similar setup as used in [Kapishnikov et al., 2019]. However, instead of using compressed image size as a proxy information level, we directly computed the entropy of the gray version of the saliency-focused image (test image).
We show the quantitative evaluation results in Figure 10. We started with a complete defocused image and gradually added salient pixels to form a saliency-focused image. We measure the entropy of the saliency-focused image relative to the complete blurred image and call it a normalized entropy, which forms the x-axis. We feed saliency-focused images to the original model and report median softmax scores and accuracy calculated over all images, which forms the y-axis. We show the evaluation procedure in Figure 2.  [Szegedy et al., 2016] and ResNet101 [He et al., 2016] on ImageNet test and validation images respectively Right: Evaluation using Accuracy Information Curve (AIC) metric for the same models and datasets.

Conclusion
In this paper, we propose an interpretability method called GGIG that leverages the model's dynamic behavior in parameter space to enhance the quality of the feature attributions. GGIG starts with a linearly interpolated path as IG and uses gradient ascent to explore how the model interacts with input in the neighborhood. We believe that by looking into the parameter space from different interpolation points, GGIG captures important information about model vs. input interaction. We show the effectiveness of the proposed method through visual inspection and several quantitative metrics across some popular architectures and datasets. As demonstrated, we expect that this work provides insights toward building a more robust interpretability method through model and input interaction.
A Quantitative Evaluation of GGIG on MNIST Dataset Figure 11: Top: Selected maps for MNIST samples generated using different methods. Bottom: Quantitative evaluations using different correlation and similarity metrics. It is obvious that the maps produced by GGIG have higher structural and numerical correlations with the input. Precisely, GGIG, as an explanation method, performs the best by a large quantitative margin.

B Saliency Maps (ImageNet Dataset)
B.1 Architecture: ResNet-101 Figure 12: Explanations (saliency maps) of ResNet-101 predictions for ImageNet samples generated by GRAD, IG, and GGIG methods. As we can see, the GGIG maps have strong correlation and similarity with the corresponding input images. Figure 13: Visualization of saliency maps obtained from Resnet-101 using 'gray' color scheme.
B.2 Architecture: Inception V3 Figure 14: Visualization of saliency maps using 'inferno' color scheme for selected ImageNet samples. It is evident that the GGIG method consistently captures the useful concepts as used by the model for predictions. Figure 15: Visualization of saliency maps using 'gray' color scheme for selected ImageNet samples. Figure 16: Visualization of saliency maps for selected ImageNet samples using 'coolwarm' color scheme. Figure 17: Visualization of saliency maps for selected ImageNet samples using 'Reds' color scheme. Figure 18: Visualization of saliency maps for selected ImageNet samples using 'reverse bwr' color scheme.


s2fieldsofstudy:
Computer Science
Physics

text:
Notes on a 1-dimensional electrostatic plasma model

A starting point for deriving the Vlasov equation is the BBGKY hierarchy that describes the dynamics of coupled marginal distribution functions. With a large value of the plasma parameter one can justify eliminating 2-point correlations in terms of the 1-point function in order to derive the Vlasov Landau Lenard Balescu (VLLB) theory. Because of the high dimensionality of the problem, numerically testing the assumptions of the VLLB theory is prohibitive. In these notes we propose a physically reasonable interaction model that lowers the dimensionality of the problem and may bring such computations within reach. We introduce a 1-dimensional (1-D) electrostatic plasma model formulated in terms of the interaction of parallelly-aligned charged disks. This model combines 1-dimensional features at short distances and 3-dimensional features at large distances.

Introduction
The construction of a charged-disks electrostatic model is motivated by the aim [1] to verify numerically the validity of the procedure which is adopted, see e.g. Ref. [2], in order to solve the BBGKY hierarchy [3] in a weakly coupled plasma when deriving the kinetic plasma equation, i.e. the Vlasov equation. Such a verification might be based on the numerical integration of the time evolution of the two-point particle distribution function as obtained from a proper Hamiltonian truncation of the the BBGKY hierarchy at the level of the three-point distribution function [4]. In 3-D space the equation for the time evolution of this two-point distribution function would be 13-dimensional (one time dimension plus the 12-dimensional two-particle phase space). In 1-D space the equation for the two-point (actually two foil) distribution would be 5-dimensional (time plus 4-dimensional two-foil phase space). However in 1-D the electric field generated by a charge foil does not decay with distance so that the two-foil interaction energy diverges at infinity. This makes it impossible to define a finite correlation length. A possible compromise is to consider the interaction of aligned, uniformly charged disks, a sort of a single-row abacus made of thin diskshaped beads, where we can introduce a characteristic length b, the radius of the disk, which we may identify with the Debye radius λ d (same for the electrons and ions). In this case the electrostatic interaction energy between two charged disks is finite both at zero and at infinite distance ξ from the source and its absolute value is a decreasing function of ξ.
The present notes are dedicated to the derivation of the main properties of such a disk-plasma in order to clarify where and how the dynamics of such a model system can mimic the dynamics of a real plasma, at least as long as the longitudinal electric field limit is concerned. In analogy with a real particle plasma, in the derivation which follows we will introduce concepts such as positive and negative charged disk densities, collective electric field shielding, disk waves etc., but we will not use a Poisson type equation with the charge disk density as the source. It will be enough to refer to the expression of the interaction energy between the disks derived in Ref. [5] that in the present model will play the role of the inter-particle Coulomb energy in a real 3-D plasma.

Disk Model
The "disk model" describes a globally neutral system consisting of a large number (as calculated inside intervals with length of the order of the disk radius b) of aligned, infinitely thin disks with equal radius, as sketched in Fig.1, equal or opposite electric charge and different masses (ion-disks, electron-disks) that are free to move along the x axis (the planes of the disks are orthogonal to the x axis) and to pass through each other. In the following, for the sake of simplicity, the ion-disks are taken to be immobile. We denote the mass of an electron disk by M e and its charge by Q e = −Q i with Q i the charge of an ion disk.

Discrete disks: interaction energy
The interaction energy W(x 1 , x 2 ) between two uniformly charged infinitely thin disks of radius b located at x 1 and x 2 respectively can be written in c.g.s. units as [5]. where and ϕ 1 , ϕ 2 are the potentials generated by the disks 1 and 2 respectively. Here ξ(x 1 , x 2 ) = |x 1 − x 2 |/b, Q 1 = πb 2 σ 1 and Q 2 = πb 2 σ 2 are the (fixed) charges of the two disks, σ 1,2 are their surface density and E(−4/ξ 2 ) and K(−4/ξ 2 ) are elliptic integrals [6]. The "disks-electrostatics" is illustrated in more detail in Appendix A. A plot of V (ξ) is given in Fig.2 . The corresponding electric force F Q 2 acting on the disk 1 due to the disk 2 is given by A plot of −V (ξ) is given in Fig.3

Continuous limit
Define an electron-disk density n e (x, t) (number of disks per unit length) and an ion-disk density n i (x, t) = n i (for simplicity stationary and homogeneous). This defines a disk-charge density (charge per unit length) ρ(x) = Q(n i − n e (x, t)), Define the electron-disk velocity u e (x, t) that satisfies the continuity equation Assuming that n i −n e (x, t) vanishes (sufficiently fast) at ±∞, the electrostatic potential created by a disk-charge density can be written in convolution integral form as which returns for ρ(x ) = Qδ(x − x 1 ) the expression for the potential of a disk given in Eq.(1). The corresponding electric force acting on a disk of charge Q j (j = i, e) can be written as which returns Eq.
. To close the system we may supplement the continuity equation (5) with the momentum equation of a "cold" disk plasma In the rest of this section we find it convenient to use dimensionless dynamical quantities based on a characteristic time (the inverse of the disk plasma frequency 1 , see Eq. (33)) where n 0 is the (uniform) background density, and on the length d (note that the disk electrostatic, contrary to standard electrostatic, has a characteristic spatial scale). Specifically we set in agreement with the definition of the variable ξ. Then we define with

Laplace transform representation
Start from Eq.(6) with the potential ϕ( where J 1 is a Bessel function and we used Eq.(46). The double-sided Laplace transform in dx can be written as Thus we obtain Computing the integral over s first would bring us essentially to the initial Eq.(6). On the contrary it is convenient to use the fact that Eq.(16) contains a special combination of Poisson transforms of the density ρ(x) and expand ρ(x) on a basis with elements that transform in a simple way under Poisson transform. Thus we write the Fourier series (finite domain, periodic boundary conditions) use the relationships and obtain Thus finally we have where the last integral is a known function (see Eq.(47.)

Fourier space
Assuming the required boundary conditions at infinity, we writê and obtain from Eq. (7), which has the form of a convolution integral, whereV (kb) is defined (without including the factor 1/(2π) 1/2 ) bŷ 24) and is explicitly calculated in Appendix A.

Screening at thermodynamic equilibrium
In order to determine the screening of a test disk of charge Q t at x = 0 produced by the spatial rearranging of the other disks we consider an electron-disk Boltzmann equilibrium 2 Note that in this disk model, contrary to a Coulomb plasma, the approximation Qϕ s /T e 1 remains valid even at close distance. Setting n i = n e (o) = n o we can write the charge density of the test disk plus the screening density obtained from Eq.(25) which inserted into Eq. (6) gives the integral equation Since the integral term in Eq.(27) is a convolution product, Eq.(27) can be solved by performing a Fourier transform with respect to x (see Appendix A for analytical details of the functions involved). Note that the factor in front of the integral is dimensionless and can be interpreted as ∼ b 2 /λ 2 d ∼ 1 if we recall that n o is the number of disks per unit length and that the disk radius is b ∼ λ d .
The Fourier transform of Eq.(27) readŝ wherê The definition ofV (kb) corresponds to (2π) 1/2 times the Fourier transform of V (X). From Eq.(28) we obtain It is not easy to compute the inverse Fourier transform (31) in explicit analytic form. Taking into account thatV (kb) ∝ 1/(kb) 2 for large (kb) and if no additional singularities inV (kb) are present 4 aside for a logarithmic term starting from the origin (see Eq.(48)), the standard procedure, would compute the integral in Eq.(31) by using a Cauchy contour of the type sketched in in Fig.(5).
The residue at the complex poleV (kb)/Θ = −1 would lead to an exponential shielding analogous to the Debye shielding for a particle plasma. The integral along the two sides of the logarithmic branch-cut would lead to a non exponential contribution not present in a particle plasma.
These qualitative statements are confirmed by integrating Eq.(31) numerically for different values of the dimensionless temperature Θ. In Fig.6 we show the result of this integration for Θ = 2. In agreement with the potential V (ξ) in Eq.(2), the shielded potential Φ s (X) is finite and nearly constant at X ∼ 0 as shown in the enlarged plot in the right frame. For larger values of X the potential Φ s (X) has a fast exponential-type decay. Afterwards, as already visible in the central frame, the spatial dependence of the screened potential changes to a slower power-like decay, as shown by the full plot in the left frame. The exponent of this power law decrease is marginally larger in absolute value than 1, i.e. marginally faster than the non-shielded potential.

Cold disk plasma waves
We consider the case where the disk thermal motion can be neglected and derive the linear dispersion relation of the cold disk plasma waves using Eqs. (5,6,7,8). Linearization for a monochromatic wave in a stationary equilibrium with density n 0 and immobile ion-disks yieldsũ which gives We see that the disk-plasma waves are dispersive for finite values of kb. In the short wavelength limit kb 1 the waves become Langmuir waves (see Note 1 and Eq.(49), Appendix A) with frequency ω Dpe where n o /(πb 2 ) has the dimension of a volume density. In the long wavelength limit (see Appendix A, Eq.(48) which defines the constants C 1 , C 2 ) we find (logarithmically corrected) "cold electron-disk sound" waves of the form We note that in a particle plasma there is no velocity directly corresponding to v D . The phase velocity of Langmuir waves is smaller than v D as follows from ω 2 /k 2 = ω 2 Dpe /k 2 = v 2 D /(kb) 2 << v 2 D while in the case of the cold sound waves 5

Disk Vlasov equation and waves
The treatment of the Vlasov kinetic equation follows standard lines with two minor differences: the expression of the interaction potential V (ξ) is different from the Coulomb potential and it is more convenient to use the particle density as the independent variable instead of the potential Thus we have the integro-differential equation is the distribution function of the j species (j = e, i), v is the disk velocity coordinate in phase space and F j (x, t) is defined in Eq. (7)] and

Linearized equation
In a homogeneous disk plasma the linearized Vlasov equation takes the form where f j0 (v) andf j (x, v, t) are the equilibrium and the perturbed distribution functions, v is the disk velocity andF is the perturbed force, as obtained from Eq.(23). Considering again for the sake of simplicity immobile ion disks and waves of the form we obtain the dispersion relation where the velocity integral has to be taken according to the Landau prescription. In the cold limit when ω/k v the , with v the the halfwidth of the distribution functionf eo , Eq.(40) becomes 5 Note that in the limit of planar foils instead of disks, i.e. for b → ∞ these sound waves would not be present.
which coincides with Eq.(33). The dispersion relation given by Eq.(40) depends on the normalized wave number kb and in addition on the equilibrium dimensionless parameter (v the /v D ) 2 = Θ (see below Eq.(30)). For disk-Langmuir waves the cold limit ω/k v the implies v the /v D 1/(kb) 1, whereas for cold sound waves it implies the less restrictive condition v the /v D | ln (kb) 2 | where | ln (kb) 2 | 1 . Normalizing velocities over v D , Eq. (40) can be rewritten in terms ofω = ω/ω Dpe ,v = v/v d and κ = kb as where the dimensionless distribution functionf eo (v) = v D f eo (v)/n o depends on the dimensionless temperature Θ = T e /(4Q 2 n o ).

Conclusions
In order to model collective electrostatic effects on a 3-D plasma using a reduced dimensionality configuration, it may be convenient to define an effective interaction energy constructed so as to retain the physical effects that we deem must be included in order to to construct a "faithful" model. In these notes we have introduced a model of interacting charged disks that provides a onedimensional model with an interaction energy that vanishes at infinity. Clearly this approach is restricted to the case where the system evolution may be described by "scalar one-dimensional electrodynamics", i.e. when no magnetic fields are present and the electric field is the gradient of a one-dimensional scalar potential.

A Disks electrostatics
A set of relevant formulae derived in Ref. [5] are listed below along with some notational adaptations.
for (kb) 2 1. In this latter limit we recover the standard Coulomb-like dependenceV ∝ 1/k 2 . The logarithmic dependence at small wavenumbers arises from the 1/x dependence of the interaction energy at large distances in 1-D.


Hornet 40: Network dataset of geographically placed honeypots

Deception technologies, and honeypots in particular, have been used for decades to understand how cyber attacks and attackers work. A myriad of factors impact the effectiveness of a honeypot. However, very few is known about the impact of the geographical location of honeypots on the amount and type of attacks. Hornet 40 is the first dataset designed to help understand how the geolocation of honeypots may impact the inflow of network attacks. The data consists of network flows in binary and text format, with up to 118 features, including 480 bytes of the content of each flow. They were created using the Argus flow collector. The passive honeypots are IP addresses connected to the Internet and do not have any honeypot software running, so attacks are not interactive. The data was collected from identically configured honeypot servers in eight locations: Amsterdam, Bangalore, Frankfurt, London, New York, San Francisco, Singapore, and Toronto. The dataset contains over 4.7 million network flows collected during forty days throughout April, May, and June 2021.

© 2022 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) Table   Subject Computer Science Specific subject area Honeypots network cybersecurity attacks Type of data Binary Table  Figure How the data were acquired

Specifications
The network flows were collected by running Argus network flow collectors [1] on the network interfaces of geographically distributed cloud servers. Data format Raw Filtered Parameters for data collection Argus network traffic flow collectors run directly on the network interface of each honeypot and store all the received traffic into files. Administration connections were filtered out by deleting the traffic with the IP addresses used for administration. Description of data collection The network flow data was collected for forty days, from April 23rd, 2021 to June 1st, 2021, using the Argus network traffic flow collector previously installed in each honeypot. Argus captures all the packets on the network interfaces of the honeypots to generate the flows. During this time, approximately 4.7 million flows were captured in total. The network connections used to administer the honeypots were filtered out from the dataset. Data source location The data was collected from eight cloud servers used exclusively as honeypots from the Digital Ocean [2]  Value of the Data • The Hornet 40 dataset contains a unique collection of network flows, considered attacks from geographically distributed honeypots. It is useful for statistical and behavioral analysis of identically configured honeypots located in different geographical locations and regions. • The data can benefit security practitioners, machine learning researchers, and statisticians working on network analysis, cyber security, or threat intelligence. It may also benefit network providers and antivirus companies. Data can be used to track attackers, to understand their origin, to understand changes in attacks patterns. • The large amount of network flows can be used to evaluate and develop better attack detection mechanisms. • The geography factor can be further used to evaluate better location placement of servers to reduce the number of attacks received by production servers.

Data Description
The Hornet 40 [3] dataset contains forty days of raw flow data, captured from eight cloud Linux passive honeypot servers. In this paper 'raw flow data' refers to network flows generated by a network flow collector from raw network data. Raw network data are the packets in the network.
Each honeypot was located in a different city in the regions of North America, Asia, and Europe. The cities were chosen from the available locations by the cloud provider Digital Ocean [2] . No honeypot software was running on the Linux servers, but each IP address received connections from the Internet.
The raw flow data was captured using the Argus network flow collector from April 23rd, 2021 to June 1st, 2021. The dataset contains a total of 4,758,657 bidirectional flows, generated by a total of 266,678 unique source IP addresses. A bidirectional network flow is an aggregation of features of all the packets in a network connection. It aggregates packets from both directions: from source IP to destination IP, and from destination IP to source IP.
The Argus flows have 118 features including 480 bytes of the content of the connection in both directions. Argus generates one binary flow file per day per honeypot. These binary files contain all the flows for that day. From this binary flow file, two more ASCII files were exported in CSV format, one with NetFlow v5 features, and other with up to 118 Argus features.
The Hornet 40 dataset consists of three main groups of files: one for the binary bidirectional Argus files (biargus), one for the ASCII files with NetFlow v5 features, and one for the ASCII files extended with up to 118 Argus features. Each of these groups is separated per honeypot, each separation per honeypot containing 40 files, one per day. The folders and file structure of the Hornet 40 dataset is as follows: All source IP addresses (from now on Src IPs) communicating with the honeypots are considered attacking IPs, due to one of the definitions of honeypots: since a honeypot is not an authorized production service, nobody should connect to it, and therefore all connections are considered attacks [4] . The main difficulty of this definition is how to consider the scanning activities of companies and organizations mapping the Internet for probable benign purposes. In this paper we still consider these benign scans as attacks, since (i) they are unsolicited, or (ii) the data collected may be sold, or (iii) used by attackers later, or (iv) use in marketing campaigns and advertising. Moreover, it is not technically feasible to filter out some of these scans based on organizations names or IP ranges, especially since many scans may be done from shared cloud providers.  The Hornet 40 dataset contains a total of 4,758,657 network flows, containing 11,794,754 packets and 1,254,838,795 bytes, originating from 266,678 unique Src IPs. Table 1 provides a general overview of the Hornet 40 dataset per honeypot, including its location, the total unique source IPs, the total flows, the total bytes, and the total packets. Table 2 compares the number of connections per network protocol per honeypot.
To understand how the raw flow data is distributed in time, Fig. 1 shows the hourly traffic distribution per honeypot during the duration of the capture. The number of flows is displayed in logarithmic scale.

Argus binary network flows
The raw flow data capture on the network interface of each honeypot was performed using the Argus network flow system [1] version 3.0.8.2. Argus stores raw flow data in a binary format that includes 118 features (e.g., total amount of bytes), and the first 480 bytes of the application content in each direction in the connection (to the honeypot, and from the honeypot). The number of features included in the binary Argus will change depending on the Argus version. These application content bytes are gathered by concatenating the content of the packets on each direction of each connection, until reaching 480 bytes. The full list of features is described in Table 3 . The binary format is highly convenient due to its high compression and thus low final size. The binary files can be read and processed using Argus client tools to generate any desired output. The

ASCII flows using NetFlow v5 features
The Argus binary flow files were processed to generate flow files compatible with the Net-Flow v5 standard [5] . This format is widely used, allowing to easily use and analyze data in CSV format. The NetFlow file is a format transformation from the original Argus file.

ASCII flows using Argus features
The Argus binary flow files were processed to generate files compatible with the NetFlow standard but with an extended number of features. These ASCII extended file are a format transformation from the original Argus file.
The Hornet 40 dataset contains one file for the extended flows, called hornet40-netflow-extended.tar.gz , that includes all the flows in the eight scenarios, and for each flow, it has all the 118 features described in Table 3 .

Network features
The complete list of the 118 features in the biargus and flow extended files of the Hornet 40 dataset is shown in Table 3 .

Experimental design, materials and methods
The dataset was collected from cloud server instances. The cloud server provider chosen for this dataset was Digital Ocean [2] . All cloud servers, have the same technical specifications: • Operating System: Ubuntu 20.04LTS • Capacity: 1GB / 1 Intel CPU • Storage: 25 GB NVMe SSDs • Transfer capacity: 10 0 0 GB transfer  The honeypots were created within the same hour of the same day. Once created, the servers were configured simultaneously using the open-source tools parallel-ssh and parallel-scp . Each honeypot has its own public IP address assigned, which has not changed during the data capture. The steps to create and configure the honeypots were as follows: • Create one honeypot per available region in Digital Ocean with the technical specifications listed above.
• Store the list of public IPs from the honeypots in a text file named hosts .
• Update the software repository of all honeypots simultaneously using parallel-ssh: pssh -h hosts -l root -o output/ "apt update'', where hosts contains the list of IP addresses of the honeypots, -l root specifies the SSH user name, -o output/ the folder where to store the results, and "apt update '' is the command that will be executed in parallel in the honeypots.
• Install the Argus network collector tool: pssh -h hosts -l root -o output/ "apt install -yq argus-client argus-server ''. The Argus version 3.0.8.2 was used. The Argus and rasplit tools produce raw flow data files from each honeypot. The binary raw flow data can be read using the ra tool provided in the Argus-clients suite. The command to read the files is: ra -F /etc/ra.conf -n -Z b -r 2021-04-23_honeypot-cloud-digitaloceangeo-1.biarguswhere -F specifies the configuration file to use (see Subsection Ra Configuration ), -n avoids resolving port numbers to service names, -Z modifies the status field to show TCP flag values, -r specifies which file to read from, and -sends the output to the standard output in the terminal.

SSH configuration for administration of the honeypots
The honeypots are configured to be remotely administered using the SSH protocol.

Ethics Statements
The work did not involve any human subject or animal experiments. The Hornet 40 dataset files were analyzed by more than 50 Antivirus systems in VirusTotal and as of January 2021 it does not trigger any detection.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


Convolution-Based Design for Real-Time Pose Recognition and Character Animation Generation

Human pose recognition and its generation are an important animation design key point. To this end, this paper designs new neural network structures for 2D and 3D pose extraction tasks and corresponding GPU-oriented acceleration schemes. The scheme first takes an image as input, extracts the human pose from it, converts it into an abstract pose data structure, and then uses the converted dataset as a basis to generate the desired character animation based on the input at runtime. The scheme in this paper has been tested on pose recognition datasets and different levels of hardware showing that 2D pose recognition can reach speeds above 60 fps on common computer hardware, 3D pose recognition can be estimated to reach speeds above 24 fps with an average error of only 110mm, and real-time animation generation can reach speeds above 30 frames per second.

Introduction
Researchers in the field of artificial intelligence have realized that the simulation of the way human neurons work is one of the pathways to artificial intelligence through the analysis of human brain behavior. In the field of machine vision, the extraction and discovery of the visual characteristics of the human pose itself have been one of the popular areas of research [1][2][3]. The human's own gesture characteristics are the information that needs to be acquired in many fields. For example, monitoring systems want to be able to determine whether a person has fallen or not based on the posture, digital entertainment applications want to draw the corresponding screen output based on the human posture, and virtual reality applications want to capture the human posture and generate the same action of the virtual human in the virtual world [4]. Therefore using machine visionrelated techniques to extract, compress, and apply the human pose itself is a topic that many researchers have been digging into. In this area, deep convolutional neural networks show great potential, so the introduction of related techniques to solve many complex problems in this process is in order [5,6].
The process of extracting, analyzing, storing, and applying human pose is often split into several independent steps to be performed. A typical step can be divided into three steps: 2D pose extraction, 3D pose estimation, and realtime pose animation generation [7]. Among them, 2D pose extraction is the basis of the whole system, 3D pose is the most commonly used output, and real-time human animation generation is the subsequent step after 3D pose data is obtained by many applications [8].
Research on human pose extraction has started long ago. Many studios have adopted solutions based on visual marker points or sensors. Actors are dressed in special costumes with visual marker points or sensors, and then, special systems are used to capture the position of the body's articulation points to achieve pose capture of the human body. The visual marker-based solutions often have a large number of cameras in the venue, each of which estimates the 2D pose within the current camera frame based on the visual marker points; the estimates from multiple cameras are then combined to produce 3D pose data [9,10]. The 3D pose data is then used to generate the corresponding virtual character movements in the movie. However, the solution based on visual marker points and sensors requires special costumes, and such a technology requires long lead times and special requirements for the location. For homes and public places, such technology is difficult to be implemented [11].
Therefore, there have been attempts by researchers to accomplish pose estimation of the human body using a single common RGB camera. Traditional solutions are often based on artificially designed rules and features, such as skin color features. Such features are often heavily constrained by scene characteristics such as lighting. Therefore, similar techniques are often used for simple tracking. An example is hand tracking [12]. With the development of deep learning techniques, it has become possible to use neural networks to "learn" target features. Therefore, many researchers have also started to investigate the use of deep convolutional neural networks to extract features for human pose estimation. In recent times, several technical solutions have achieved good performance [13]. A typical example is the OpenPose scheme based on deep convolutional neural networks developed by CMU.
However, the current schemes do not fully satisfy the needs of the applications. One of the reasons is that there are still many limitations in the real-time performance of these solutions. For example, the hardware of OpenPose that can achieve real-time is limited to the current high-end GPUs, and it cannot achieve the real-time requirement on the low-end GPUs [14]. Many 3D pose estimation techniques that claim to be real-time rely on high-quality 2D pose input, which takes time to obtain. The second reason is that there is no complete solution from 2D pose extraction, 3D pose, and estimation to real-time character animation generation. And many film and TV, game, and virtual reality companies need a complete solution more than anything else.

Related Work
Real-time pose recognition and animation generation are an important ongoing research direction in computer graphics, and the widely used method is deep learning. Real-time pose recognition and animation generation using deep learning are still a challenging task, and domestic and international research on this topic consists of the following three main areas.
2.1. 2D Posture Recognition. Currently, a typical scheme in the field of 2D pose recognition based on deep learning is the one based on Mask-RCNN [15]. In contrast to the two-stage scheme, all the limbs in the frame are extracted directly. Subsequently, all interconnected limbs are acquired to directly generate pose recognition results for all people, and such a scheme is called one-stage scheme. A typical implementation of this research direction is CMU's Open-Pose [16], where OpenPose first selects VGG [17] to build a feature extraction network for backbone and then uses multiple iteratively corrected refine networks to achieve the final result extraction. Of course, there are special series of algorithms in addition to this. For example, DensePose [18] represents a recognition algorithm with limb region recognition as the core. This type of algorithm does not only identify the type of a "point" but also marks the whole range of the limb area. These solutions are more complex in their tasks, and therefore, it is difficult to achieve real-time.
2.2. 3D Posture Recognition. The idea in the field of 3D pose recognition is to build on the 2D pose recognition and further estimate the 3D pose. This scheme mainly relies on the high accuracy 2D pose recognition results. For example, [19] proposed a feedforward neural network-based scheme, which directly estimates the corresponding 3D pose using a neural network based on the already extracted 2D pose. Based on a similar scheme, Facebook AI introduces the information of time series to further improve the accuracy of 3D pose. The other one is to estimate 3D pose directly using images as input. For example, the scheme proposed in [3] prepares a parametric 3D human model and then constructs an encoder-decoder network directly based on the given image and then uses this network to predict each parameter of the parametric model to achieve the pose prediction of the human body. The scheme is able to predict both human pose and human body size.

Real-Time Character Animation Generation.
Regarding the scheme of real-time character animation generation using feedforward neural networks based on a preextracted 3D pose database, the main research idea is to take the current environmental features near the character and the character's behavioral orientation (e.g., turning and jumping) data as input and train the neural network with matching pose as the desired output; then, the nearby virtual environmental features are continuously extracted and input to the neural network at runtime. Then, we extract the nearby virtual environment features into the neural network at runtime to build a highly realistic character animation. Among them, Phase-Functioned Neural Networks [8] further improve the realism of character animation by introducing forced periodicity, and this scheme is a representative research result in this direction.

Multitask Multilayer Neural Network for Real-Time 3D Pose Estimation
In this section, we first discuss the representation of the pose in the system that determines the neural network and postprocessing design. Then, we discuss each step in the pipeline. A multitasking neural network is used to output 2D pose information and 3D information simultaneously. And a multistage detection structure is used to maintain speed and accuracy. The second postprocessing step involves detecting, linking, and automatically matching the pose in 2D image space with the depth in 3D world space. In contrast to the typical 3D pose estimation system simple 3D pose baseline [6], our work uses a specially designed intermediate representation to avoid losing depth information. This design allows our 2 Wireless Communications and Mobile Computing system to perform both 2D and 3D detections in a single network, which means that we are able to achieve higher speedups and the neural network is better able to convey potentially useful information.
A typical 3D pose estimation system tends to go through the following process as shown in Figure 1.
The input image is first processed by a 2D pose recognition network to produce a 2D pose. The 2D pose is then passed to the 3D pose estimation network to be processed one by one to produce a 3D pose for each person. This process involves both 2D and 3D networks, and the second 3D pose estimation network will run more than once. This leads to the following problems.
(1) The 2D pose network converts the input image into an abstract 2D pose, a process that results in the loss of the image information carried in the original image. Therefore, it is not possible to improve the accuracy of the 3D pose estimation network based on information such as pixel values (2) In the current case, there are two networks to run, and in a practical deployment, it is often the case that the first network is started first. The first network, after running to get the output results, offloads the first network and then starts the second network. This results in a significant performance loss and limits the possibility of performance improvement In summary, traditional 3D pose estimation systems have bottlenecks in terms of further accuracy and operational speedup, so we propose our own 3D pose estimation network scheme. Instead of having two steps, the scheme outputs all the information needed to assemble the 3D pose in a single run.
This section first describes how to design a 3D pose representation that is more suitable for the output of the neural network, followed by a discussion of the specific construction of the network.
3.1. 3D Pose Representation Based on Relative Depth. The pose representation is like a bridge between the neural network and the postprocessing system. Therefore, the definition of the representation has a great impact on both neural network design and postprocessing system design. We use CNN as the basic module block, which means that the output is an image. Therefore, we define an imagebased representation for all human poses. Continuing from the 2D pose recognition system, we consider the human pose as a directed Figure 2. In this directed graph, each node in this directed graph corresponds to a human node j. Each node contains the following information: ð Class j , X j , Y j , D j , Offset X j , Offset Y j Þ. This information is assembled into a 3D pose in the manner of Figure 2.
Class j denotes the type of the current node. X j , Y j denotes the 2D coordinates of the current node. D j denotes the distance from the current node to the camera. Offset X j , Offset Y j denotes the relative offset from the parent node of the current node to the current node. This encoding continues our research in 2D pose recognition systems.
In order to simplify and speed up the process, we want to make the network task as simple as possible. Therefore, in this example, we encode the 3D joint position in two parts: The 2D position can be reused from the pose linking process. Therefore, depth D j is the only additional information needed. Based on the network output, the postprocessing execution mapping F will convert the 2D limb nodes to 3D space in the following way: We encode the depth value as the relative depth in the world space, which is to make the probability distribution of the depth values of the nodes as homogeneous as possible, thus facilitating the learning of the neural network. We consider the human pose as a tree structure, with the head as the root node. For each joint, the connected joint near the head will be the parent node, while the distant joints will be the child nodes. Then, the relative depth is encoded as the difference value of the depth of the current joint to the depth of the parent joint. We make sure that each joint has only one parent; otherwise, the relative depth is not unique. The reason why relative depth is easier to learn than absolute depth is that it is independent of body position and requires only local information. This is in line with the local field of perception characteristic of convolutional neural networks.

Wireless Communications and Mobile Computing
Obviously, there is no unique representation for 3D pose. A typical 3D pose representation is the following matrix: Each row represents the 3D coordinates of an articulation, and n is equal to the number of articulations of the current figure. This matrix representation is used by networks such as simple 3D baseline. However, this representation means that the network F can only perform the following mapping: where UV represents the 2-dimensional coordinates of the nodes in the image space.
However, our desired output is as follows: In this case, the length of the output sequence is uncertain. Therefore, it is difficult to use neural networks for modeling. An optional way is to run the network F multiple times to output all sequences: The problem with this scheme is that the number of runs of the network is uncertain, and the length of the input vector becomes longer as the length of the sequence grows. This is not conducive to the design of the network.
Another option is to divide the regions of each body in advance and then perform one-by-one mapping. This is the scheme adopted by simple 3D baseline [20][21][22].
In contrast to the design of these schemes, we have adopted a different idea. We want the mapping of the network itself F is a simple mapping: Then, add a new fast linking algorithm C that outputs the result as an infinitely long sequence: This solution effectively avoids running the neural network repeatedly and ensures the speed of operation.

Multitasking and Multilevel 3D Detection
Network. The design goal of our network is to maintain accuracy and achieve high operational speed. With this in mind, we propose and provide a new multitasking and multilevel 3D detection network architecture. Our network architecture consists of three parts' feature pyramid network, a 2D detection branch, and a depth detection branch. The depth detection branch looks almost identical to 2D detection, except for the map generation module. In each detection branch, we perform detection at different feature level and then connect them together. Finally, the map detection module analyzes the concatenated results and outputs out the final map. Figure 3 illustrates the structure of our 3D pose detection network. First, a ResNet34-like backbone processes the input image and then comes the deconvolution and connection layers. These two structures build a U-Net structure. Then, there are two separate branches. One is used for 2D detection to output 2D probability maps and 2D offset maps. The other is used for the 3D depth map. Each detection branch consists of a multilevel detection module, a connectivity layer, and a map generation module.

Multilayered 3D Inspection Network Skeleton.
Convolutional neural networks usually have the best detection target size. Therefore, in order to accommodate different target Wireless Communications and Mobile Computing sizes, the usual approach is to process the input image multiple times with different scaling. However, in our case, processing the input image multiple times would take a lot of time. Instead, we use a multilayer architecture based on a feature pyramid network to perform detection at multiple scales through the network structure. This structure can adapt to the size of the target without adding too much to the computational cost. Since the computational cost savings are redundant, this data can be preserved without causing accuracy loss. For each region in the input image, there is only one optimal detection level, which means that no other detection levels need to be computed. The network structure of the multiscale feature pyramid has already been discussed in the context of the 2D pose recognition algorithm, so again we will not dwell too much on it [16].
Here, we only describe the parts of the 3D pose estimation network structure that differ significantly from the 2D   In detail, we use ResNet34 as the frontend but compress the channels to half. The detailed structure is shown in the table. After this, there is an additional convolutional layer. Then, there are three pairs of deconvolution layers and the connection layers.
3.4. Design of the Loss Function. As a multitask neural network, we need to carefully design the loss function to balance each different branch. Otherwise, one branch may be weaker than the others.
In our case, for the classification branch, we directly use the L2 loss instead of the maximum cross-entropy loss. Then, for the other two branches, we focus only on the locations with joints, not the background. For the background region, we directly ignore the loss and allow the branch to output any result. We find that this makes the task easier, and the network can achieve lower errors.

Data Set and Labels.
In order to train the generative model in this paper, we collected about 25,000 headshots from the Web, from different people and with different resolutions. For the interception of people, we use the open source lbpcas-cade animeface script to locate and intercept the faces of anime characters from large images. In line with our purpose of image generation, the model in this paper is mainly applied to the avatars of personal information pages, etc. Therefore, these images are uniformly scaled to 64 × 64 and the higher resolution version 96 × 96 for training the network in this paper.
Since the generative model in this paper has an auxiliary classifier, it is necessary to classify the images in the real dataset. We found that when using the original WGAN [4], LSGAN [18], DCGAN [3], etc. for image generation, the hair of the person is blended with the background, and the boundary is not clear, resulting in poor visual realism, so the model in this paper uses hair color as the class of the image, in order that the model can learn how to "draw hair" to increase the realism of the image.
Based on the actual data set and the theory related to CIEDE2000 color difference calculation, we finally determined 8 categories of colors, which are black, white, red, green, blue, brown, purple, and yellow. Among them, red, green, blue, and yellow are the four colors with 90°color phase angle separation on the LAB color space, while the two colors brown and purple are 45°apart from yellow, green, red, and blue, respectively, on the color phase diagram. By intercepting the hair part in the sample to get its color, the CIEDE2000 color difference theory is used to calculate the class to which the color should most belong to determine the class of the image, so as to get the label.

Generating Diversity and Resolving Pattern Collapse.
Pattern collapse is the process in which the generative model converges to a point in the output space incorrectly in order to achieve the goal of deceiving the discriminator, i.e., generating an almost identical image, and even this image may be very unrealistic. This problem is better solved by the model in this paper on the collected data set. Figure 4 shows the generation of training images with different models at different epochs. The images generated by this model are more aesthetically pleasing to human vision than other models in general.
We find that these models can generate images normally in the early training period, but after the training epoch reaches 100 generations, both DCGAN and LS-GAN are prone to pattern collapse, as shown in Figure 5. In contrast, this model solves the pattern collapse problem to a certain extent and improves the "realism" of image generation.
To verify the reliability of our model on this dataset (less prone to pattern collapse), we counted several different GAN models and conducted 10 experiments at the maximum number of iterations we set (at which the model should have converged), and the number of pattern collapses occurred as listed in Table 1.
Among them, the WGAN experiment uses the RMS optimizer and sets the learning rate to 0.002; DCGAN, LSGAN, and ACGAN use the Adam optimizer and sets the learning rate to 0.0002, while our LMV-ACGAN uses the RMS optimizer and sets the learning rate to 0.002 for the optimization of G and D___CNN parameters and uses the Adam optimizer and sets the learning rate to 0.0002 for the threeMLP networks with Adam optimizer and learning rate set to 0.0002, beta1 to 0.8, and beta2 to 0.98.
For generative diversity, we use the FID (Fréchet Inception Distance) metric to judge the goodness of the model. x denotes the distribution of the real sample set, g denotes the  Figure 6: Comparison of classification loss. 6 Wireless Communications and Mobile Computing distribution of the generator output, μ denotes the mean of the distribution, Σ denotes the variance of the distribution, and Tr denotes the trace of the matrix, and the FID is performed according to equation (8) calculation.
where μ x and μ g denote the mean values of the features of the real image and the generated image after extracting the intermediate layer by the same inception network, and Σ x and Σ g denote the variance of the extracted intermediate layer feature values. Lower FID implies higher quality and diversity of images.
Since FID is more sensitive to model collapse and more robust to noise, this distance score of FID will be quite high if there is only one image. Therefore, FID can be used to describe the diversity of GAN networks.
In this paper, we experimented with several GAN models in Table 2 and obtained the generator model by training 100,000 batches without pattern collapse and generated 25,000 samples as the distribution sampling of g by this model and calculated the FID distance from the real data set as the FID score of this generator model.
From the experimental data in Table 2, it can be analyzed that providing labeled data on this generation task can be very effective in improving the generation quality of the generative model and at the same time can effectively reduce the probability of pattern collapse occurrence. At the same time, the model in this paper can better ensure the diversity of image generation. Figure 6 compares the changes of auxiliary classifier losses during training between the original ACGAN and our method. From the figure, it can be seen that although the increase of hidden parameters and multiscale discriminations lead to the increase of model parameters, the increase of parameters does not affect the convergence speed of this model excessively. At the same time, due to the addition of the discriminant information, the mean value of classification loss decreases from 2.94 to 2.81 after 35,000 iterations, which improves the classification ability of the auxiliary classifier to a certain extent, and this also improves the accuracy of the generator G in generating images by category. At the same time, this model can achieve better discriminative results with fewer iterations than ACGAN.

Conclusions
In this paper, two new neural network structures are designed for 2D and 3D pose extraction tasks, and the corresponding GPU-oriented acceleration schemes are given. Experimental results show that the pose recognition and animation generation system proposed in this paper achieves the set speed and accuracy goals with an average error of only 110 mm, and real-time animation generation can reach a speed of more than 30 frames per second. It demonstrates the successful application of this paper in the field of computer graphics based on deep learning techniques.

Data Availability
The experimental data used to support the findings of this study are available from the corresponding author upon request.


extfieldsofstudy:
Computer Science
Engineering

text:
On-Sensor Binarized Fully Convolutional Neural Network with A Pixel Processor Array

This work presents a method to implement fully convolutional neural networks (FCNs) on Pixel Processor Array (PPA) sensors, and demonstrates coarse segmentation and object localisation tasks. We design and train binarized FCN for both binary weights and activations using batchnorm, group convolution, and learnable threshold for binarization, producing networks small enough to be embedded on the focal plane of the PPA, with limited local memory resources, and using parallel elementary add/subtract, shifting, and bit operations only. We demonstrate the first implementation of an FCN on a PPA device, performing three convolution layers entirely in the pixel-level processors. We use this architecture to demonstrate inference generating heat maps for object segmentation and localisation at over 280 FPS using the SCAMP-5 PPA vision chip.

Introduction
Fully convolutional neural networks (FCN) have been used across many modern computer vision tasks such as object detection [10], classification [15] and segmentation [19,26]. However, the deployment of deep FCN usually relies on powerful GPU/CPUs which are typically not present in emerging embedded edge devices, where cost and energy considerations dictate stringent limits on storage and computing resources. Despite this, there is an ever increasing demand for artificial intelligence on such edge devices. One promising approach to the edge computing hardware is represented by Pixel Processor Arrays (PPA). Unlike conventional vision systems, which consist of separate sensing and computing hardware, PPA devices are emerging vision architectures, integrating sensing, storage, and computing on a single silicon chip ( Fig.1) [4,14]. Such integration optimises data movements in the system, promising high performance and low-power consumption, but requires careful algorithm implementation, in order to efficiently utilise the hardware resources available in an on-sensor Text Text (a) (b) Figure 1: (a) Hardware architecture of the Pixel Processor Array (PPA) used in this work. The SCAMP-5 camera is based on a PPA vision chip with 256×256 Processing Elements (PE), which are simple programmable processor cores where parallel image processing is conducted by directly operating on analogue signals (e.g. electric current from photodetector PIX, which is proportional to the light intensity) within analogue registers (AREG) and bit operations within digital registers (DREG). (b) The ultra-compressed sensing capabilities with ×32,000 data reduction and a low operation power on PPA using the proposed FCN for 2D localisation.
computing device. This paper demonstrates how to implement and deploy binarized FCNs on PPA hardware. Networks with binary weights and activations are difficult to train and usually suffer from performance drop when compared to their floating-point equivalent. Use of batch normalisation has been proposed to avoid gradient explosion and train binarized neural networks successfully [30]. In this work we introduce batch normalisation into binarized FCNs to improve the training efficiency and the performance of inference on PPA arrays. We implement a purely binary convolutional network containing both binarized weights and activations. The use of binary activations alleviates accumulative errors introduced by approximate computations used to perform image convolutions upon PPA hardware devices [12]. This error mitigation allows our approach to perform deeper networks than previous work [2,3,21] wholly upon the focal plane without encountering an increasing loss of accuracy that would occur otherwise. Furthermore, it is noted that the implementation of the batch normalisation, the sign activation function, and learnable activation threshold for binarized activations is equivalent to adding a bias matrix to the layer activations, which significantly simplifies the inference process on sensor. With this binarized FCN, the inference calculation process can be implemented entirely with efficient add/subtract, threshold, and shifting operations for all layers. This scheme specifically benefits PPA computing devices such as the one shown in Fig. 1 because it matches the simple instruction set of these devices, and reduces the impact of calculation errors caused by noise accumulation when using analogue registers (AREG) for storage and calculations, especially for activations. For experiments, we train binarized FCN and deploy it on the PPA for object localisation and coarse segmentation. On-Sensor Computing and the SCAMP-5d Vision System: The concept of on-sensor computing originates from emerging novel circuit designs that enable direct signal processing on the sensing chip [33]. The SCAMP vision sensor [6] used in this work is based on a PPA concept implemented using mixed-signal analog/digital datapath (Fig. 1), other devices such as SONY IMX500 1 or Aistorm Mantis 2 integrate sensing and computing resources in a single device using alternative strategies. Our work takes advantages of the SCAMP analog/digital PPA to efficiently implement a binarized FCN. As shown in Fig. 1, each sensing element converts light into analogue signals that are immediately processed on the focal plane. Unlike the hardware architecture of standard computer vision systems, the PPA does not involve Analogue-Digital-Conversion (ADC) after sensing. Instead it directly operates on analogue electric currents, in physical proximity to the image sensor pixel circuits, accelerating the signal processing and avoiding the bottleneck of ADC and data transmission process to external processor units. However, such analogue processing introduces errors into the computation and data is prone to noise and temporal decay [12]. Fig. 1b shows the data extraction capabilities of the PPA based on the proposed FCN, effectively reducing images of thousands of pixels to specific contextual information within a tiny number of bits. The SCAMP vision sensor has been demonstrated in several applications in the field of robotics [17,18,22,28] and computer vision [1,5,27]. Contributions: The main contributions of this work are: 1) We propose, train, and demonstrate the use of a purely binarized network (both binary weights and activations) specifically for PPAs. This approach of binary activations addresses the accumulation of analogue computing errors and value saturation after each layer, thus enabling deeper networks while maintaining performance. 2) We present the first implementation of an FCN architecture for PPAs. Our approach uses group convolutional layers [31], and stores hundreds of convolutional filter weights upon the focal plane of the PPA. 3) Unlike earlier work, we apply batch normalisation during training and utilise this to learn bias parameters to be applied during inference on the PPA device. 4) We provide the first demonstration of object localisation and coarse segmentation tasks on a PPA, with previous works being only concerned with classification tasks. In addition, we created a simulated environment and datasets (to be available after publication) to support PPA developers and researchers for idea validation off and on-sensor.

Related Work
Recent work for PPAs has concentrated on CNNs and demonstrated on classification tasks [2,3,21]. However we found no previous PPA work on FCNs which are important for further tasks like localisation and segmentation. The research on CNN implementation and inference within PPA was pioneered by Bose et al. [2] where a CNN with a single convolutional layer was implemented upon the PPA array and a fully-connected layer upon its controller chip. Their work performs 16-bit image convolution operations using 4×4 DREG "Super Pixel" blocks and demonstrates live digit classification based on MNIST dataset at speed of around 200 frames per second (FPS). In their work, the ternary {-1, 0, 1} kernel filters are stored in the program memory, and are effectively encoded in the instructions/operations sent to the PPA array, performing convolutions sequentially. To fully take advantage of PPA's parallel computing characteristics and further improve the CNN inference efficiency, Bose et al. [3] proposed the idea of in-pixel weight storage, where the network's weights are directly stored within the registers of the PPA's processing elements. This enabled both parallel computation of multiple convolutions, and implementation of a fully connected layer upon the PPA array resulting in a ×22 faster CNN inference (4464 FPS) on the same digit recognition task. Based on these two works, Liu et al. [21] proposed a high-speed lightweight neural network using BinaryConnect [8] with a new method for computing convolutions upon the PPA, allowing for varying convolutional stride. Their work demonstrated four different classification tasks with frame rates ranging from 2,000 to 17,500 FPS with different stride setups. Based on their network, a direct servo control using real-time CNN inference results [23] and a simulated robot tracking from a drone [24] with on-sensor CNN computing results are presented. Moreover, AnalogNet [32] implements convolution with value approximation for multiplication on the focal plane using 3 kernels and fully-connected layer on the micro-controller based on the MNIST. AUKE [11] is developed to automatically generate convolution kernel code on the PPA by Debrunner et al. Our work advances state-of-theart in neural network implementation on PPA. We propose a new FCN network architecture suitable for PPA implementation, including three convolutional layers with binary weights and binary activations. The network utilises group convolution and is trained using batch normalisation, and enables new segmentation-related applications.

Method
Neural network architectures for PPAs must be carefully designed taking into account model size, architecture, and the feasibility of exploiting the PPA's parallel computation and onsensor storage. This is essential due to the limited on-sensor resources compared to standard computer hardware which may have access to powerful GPU/CPUs. This section attempts to find a balance between the neural network performance and its efficient implementation on the PPA.

CNN with Binary Weights and Activations
BinaryConnect [8] trains neural network with binary weights, however, it is not a fully binary neural network with floating-point neuron activations. The neural network in our work is based on Binarized CNN [9], with binary weights and neuron activations, which can be stored and processed with bit-wise operations. Compared to BinaryConnect, Binarized CNN reduces the intermediate memory storage for activations and replaces most arithmetic operations with bit-wise operations. Such fully binarized networks are thus highly suitable for PPAs due to their small memory footprint. For the forward propagation, we take different strategies to binarise the weights and activations to simplify the binarization process. All the weights are binarized with a deterministic function Eq.1 where w r is floating-point weights and w b is the binarized weights. In terms of activations, we train channel-wise learnable thresholds α to binarise the activations to obtain more informative binary feature maps, inspired by work [25]. Additional coefficients, introduced by channel-wise thresholds, have low impact on the implementation efficiency on the PPA. In Eq.2, α is the trainable thresholds for binarization of each channel, a r is the real-valued activations and a b is the binarized activations. During the training process, using standard backpropagation and stochastic gradient descent, the gradients are calculated with the floatingpoint weights. The weights and activations are only binarized during forward pass. In our work, the Binarized CNN is trained on a PC machine and the CNN inference process is implemented on the SCAMP-5d vision system.
The training process for batch norm parameters can be seen from [20] Algorithm 1, in which ε is used to avoid a zero denominator and the main scaling and shifting parameters γ and β for batch norm are learned during the training process. Then the batch norm can be applied to manipulate activations [20]. In Fig. 2   forward propagation process: Considering activation function tanh does not change the sign of inputs, we have: In Equation 6, where β , γ, and α are all trainable parameters that can be obtained directly after training. Thus the 'bias' B can be calculated used these parameters offline, before implementing it on the PPA. During the inference process, the batch norm and activation reduces to a bias term, as shown in Equation 5 B on Y. Hence, the inference process on the PPA can be simplified as shown in Fig. 2b. 3.2 FCN architecture on sensor FCN is a CNN architecture providing pixel-level classification, targeting image segmentation [26]. This paper proposes a 3-conv layer FCN that can be implemented on a PPA sensor. This work extends from previous CNN classifications, that were done using fullyconnected output layers [3,21]. In this paper, FCN is used for heat map generation by adding one convolutional layer with 128 filters and replacing the final fully-connected layer   with a convolutional layer of kernel size 1×1. Fig. 3 shows the overall FCN architecture with configurations for each layer. Fig. 6 illustrates the first convolution layer, generating 16 binary feature maps. The second layer adopts group image convolution [7] of 8 on the input 16 feature maps to make a trade off between convolution computation complexity and network performance, where each of 64 outputs is generated by adding two intermediate feature maps (Fig. 9). These 64 binary feature maps from the second convolutional layer are stored in 4 DREG. The third layer then generates the final heat map representing the prediction probability distribution, taking these 64 binary feature maps and combining them within an AREG. Each binary feature map being multiplied by an associated weight of -1/1.

FCN deployment on the PPA
This section gives the implementation detail of the binarized FCN on the PPA sensor hardware. First Layer: 16 binary filters are replicated to fill a DREG (Fig. 7) for parallel convolution purpose [3,21]. Fig. 5 shows the image convolution process on the PPA. The image convolution on the PPA can be decomposed as 'multiplications', shifting and addition and the convolution result is obtained by performing shifting and addition process for 16 times with a stride = 1. Then the pre-calculated bias B is plotted into 4×4 grids on a AREG and is subtracted from the feature map (seen in Fig. 6 ). Then the output binary image is obtained by binarizing feature map after subtracting B. In this layer, tanh is used as the activation function. When implementing inference on sensor, the tanh activation function is transformed into binarisation with a sign function, with offset computed from batch norm parameters as can be seen from Eq. 4. This layer shares some similarity with work [21] including the image resize, replication, and image convolution but with extra activation binarization process. Second Layer: The key to the second layer is to implement the group convolution with 16 feature maps as inputs and 64 as outputs. By dividing them into eight groups, thus there are 128 binary filters need to be stored on sensor. The layout of filters directly affects the inference efficiency. We design a storage structure for filters in first (Fig. 7) and second layer. Fig. 8 illustrates the layout of these filters within one DREG. As can be seen, each time to perform a convolution, the corresponding kernel filters are activated in parallel, shifted, and  replicated to fill each 64×64 block in 256×256 PEs. This filter storage structure can also extend to store more filters following the similar way to fill all PEs with 16 filters. More details can be seen from Section 6. In Fig. 9, to implement second convolution layer with 8 groups, the input 16 binary feature maps are first transformed by switching position of adjacent maps. This is followed by convolution with associated 32 filters for these 32 feature maps. Then 16 gray-scale feature maps are obtained by adding each two of 32 maps. By performing convolution for another 96 filters, 64 gray-scale feature maps can be derived. The bias matrix is subtracted and then after binarization, 64 binary feature maps are generated. Third Layer: In this layer, as shown in Fig. 3, 64 1-bit filters are plotted on a DREG, followed by 'multiplication' with the 1-bit feature maps from the previous layer. After 1 × 1 convolution, these 64 feature maps in 4 DREG is relocated to 1 AREG after 2 × 2 maxpooling. The summation of these extracted feature can be obtained by shifting and adding into    Fig. 4). Unlike in the previous two layers, the activation function for this layer is ReLU to generate a gray-scale feature map as the final prediction result of the network.

SCAMP-5 Inference, Experiments, and Evaluation
This section demonstrates the application of the proposed network architecture to coarse segmentation and object 2D localisation from a bird's eye view. We implement the FCN algorithm on the SCAMP vision system hardware (Fig. 10). We set up a realistic environment in Webots 3 [29] robot simulator (Fig. 11) for data collection and the validation of FCN deployment on sensor. Training, testing and validation datasets are collected by repeatedly taking images from a flying drone equipped with a simulated "SCAMP" and then validation images are sent to the PPA hardware for inference. Binarized FCN is trained offline based on these datasets with the method proposed in Section 3. The whole neural network for both coarse segmentation and localisation is performed on sensor. Fig. 11 shows the samples of collected datasets and their annotations for segmentation of road and grass. To validate the performance of the proposed network on different tasks, a Task  IoU  Road segmentation on simulation (Computer) 74.0% Road segmentation on sensor (PPA) 69.3% Grass segmentation on simulation (Computer) 76.6% Grass segmentation on sensor (PPA) 72.9% Table 1: Intersection over Union (IoU) performance comparison between simulation on computer and execution on-sensor for coarse segmentation. Figure 13: Training process comparison for grass segmentation with and without batch norm.   road and grass coarse segmentation is explored in this section. As shown in Fig. 11, we directly use the road/grass shape as the ground truth for coarse segmentation. Notice that the trees and grass areas often share similar gray-scale levels with the road, making coarse segmentation unfeasible by simply using binary thresholding. Tab. 1 shows the Intersection over Union (IoU) performance comparison between FCN inference on simulation and on sensor. Specifically, IoU is measured here by counting the number of intersected pixels over the number of united pixels of the predictions and groundtruth. In addition, Fig. 13 compares the FCN training process for segmentation task between using and not using batch norm, which shows the binarized FCN dose not converge without batch norm, justifying its use here. Some of the results can be seen from Fig. 12. Tab. 1 compares the experimental results on sensor and its counterpart baseline on computer with identical neural networks and validation images.

Object Detection
We also implemented an object detection task, based on the heat map. As for the object localisation, rather than using the vehicle segmentation image as the ground truth for training, we use Gaussian position distribution (Fig. 11) as the ground truth since the probability distribution is adequate to represent the object 2D localisation. For the validation, a distance threshold is set from 0 to 63 to count the number of predictions with a distance to the groundtruth that falls into this threshold. A zero distance means a perfect prediction. The final localisation is obtained by the weighted sum of all the possible positions. After the test, within a distance of 10 pixels, the vehicle localisation accuracy for simulation and SCAMP is around 88% and 83% respectively (Fig. 14). Tab. 2 shows the FCN performance in terms of time, power consumption and model size.

Pupil detection
Considering the light weight and low-power consumption of the PPA chip and the increasing popularity of virtual and augmented reality (VR/AR) based on the eye movement, it is promising to mount the PPA chip to a wearable device such as glasses in the near future, hence we explore the pupil detection with the proposed binarized FCN based on the public dataset of eye images: TEyeD [16]. Fig. 15 shows some of the training images and their annotations and Fig. 16 shows results comparison between SCAMP and simulation inference, where the accuracy curve is plotted according to the Euclidean distance between simulation/scamp inference results and the groundtruth. Within a distance of 10 pixels, the localisation accuracy for simulation and scamp is around 88% and 83% respectively. More experimental results can be seen from supplementary materials. Notice that there is around 5% -6% performance gap for the experiment on sensor compared to the simulation. This is due to noise in the convolution operation performed on AREG because of the inherent non-idealities of analog computation [13] and some random bit-flipping errors observed in DREG when performing massively parallel shifting and replications. Mitigation of these issues requires further software or hardware solutions. In this work, we tried to find a balance between network complexity and viability for deployment upon the available PPA prototype hardware. Pixel-wise accurate segmentation, with a quality equal to one that can be obtained using a CPU/GPUs hardware, using embedded low-power SCAMP-5d vision system, is still a challenging task with current hardware and neural network architecture.

Conclusion and Future Work
On-sensor computing is important for embedded and low-lag, low-power vision systems. Due to their compactness and computational advantages, binary FCNs are increasingly appealing. In this paper, we propose and implement a method that demonstrates carrying out an inference with a binarized FCN on an on-sensor computing device. In contrast to previous works that have mainly focused on classification with a fully-connected layer, we, for the first time, exploit the FCN architecture design, implementation method, and inference on sensor. We validate, using a real pixel processor array (PPA) hardware, on the visual competences of region segmentation and target object localisation with a latency of 3.5 milliseconds for each inference. With the development of future generation of on-sensor devices in terms of image resolution, manufacturing techniques, and local memory capacity, we believe our proposed binarized FCN can be extended with extra layers for more challenging applications.
Researches will be able to baseline against our work here and benefit from the code and data we will release after publication.