TaylorMade VDD: Domain-adaptive Visual Defect Detector for High-mix Low-volume Production of Non-convex Cylindrical Metal Objects

Visual defect detection (VDD) for high-mix low-volume production of non-convex metal objects, such as high-pressure cylindrical piping joint parts (VDD-HPPPs), is challenging because subtle difference in domain (e.g., metal objects, imaging device, viewpoints, lighting) significantly affects the specular reflection characteristics of individual metal object types. In this paper, we address this issue by introducing a tailor-made VDD framework that can be automatically adapted to a new domain. Specifically, we formulate this adaptation task as the problem of network architecture search (NAS) on a deep object-detection network, in which the network architecture is searched via reinforcement learning. We demonstrate the effectiveness of the proposed framework using the VDD-HPPPs task as a factory case study. Experimental results show that the proposed method achieved higher burr detection accuracy compared with the baseline method for data with different training/test domains for the non-convex HPPPs, which are particularly affected by domain shifts.

Introduction
In this study, we address the problem of visual defect detection (VDD) for high-mix low-volume production of non-convex metal objects ( Fig. 1), such as highpressure cylindrical piping joint parts (VDD-HPPPs). At automatic metal processing site, when drilling holes in metal using a robot hand, defects called burrs occur. The presence of these burrs often causes scratches and cuts on the hands, which deteriorates safety and affects the accuracy of the product. As HPPPs are produced in small lots, the burr inspection process is not fully automated and demands manual effort. In some factories, the visual inspection is carried out by more than six workers, for 18 hours a day, which is laborious and costly. To address this, automatic eye-in-hand VDD presents a promising solution to this problem.
Automatic VDD on metal objects has been a long standing issue in machine vision literature [1] and has been energetically studied in various places [2,3,4,5,6,7]. In recent years, there have been attempts to use deep learning for this problem. In [8], a flexible multi-layer deep feature extraction method based on CNN via transfer learning was developed to detect anomalies. In [9], structural damage detection method based on Faster R-CNN was developed to address the issues of object size variation, overfitting, and specular reflection. In [10], remarkable progress was made in detecting corrosion of metal parts such as bolts.
Majority of these existing VDD approaches targeted convex metal objects such as flat steel surface [11]. Therefore, it is often assumed that the risk of multiple reflections is low and simple foreground/background models were used. In contrast, the HPPPs targeted in this study are non-convex objects. Therefore, subtle difference between training and test domains has a large impact on foreground appearance (i.e., burrs) and background prior.
In this study, we address the above issue by introducing a tailor-made VDD framework (Fig. 2) that can be automatically adapted to a new domain (e.g., metal objects, device, viewpoints, lighting). Specifically, we formulate this adaptation task as the problem of network architecture search (NAS) [12] on a deep object-detection network, in which the network architecture is searched via reinforcement learning. We demonstrate the effectiveness of the proposed framework by using the VDD-HPPPs task as a factory case study. Experimental results show that the proposed Figure 2. From manual to automatic VDD. method achieved higher burr detection accuracy than the baseline method for data with different domains for the non-convex HPPPs, which are particularly affected by domain-shift. The HPPPs have a wide variety of registered products, about 20,000 items. The production quantity is mainly small lots, and the monthly production starts from one. This is very different from the conventional applications such as hydraulic parts and engine parts flowing on a dedicated line [13]. Existing image recognition softwares often require as many as seven days to adjust the program to adapt to a new domain, and it is necessary to change the specifications of parts and items. Therefore, they are not suitable for small lot products such as HPPPs.

Visual Defect Detection (VDD) Problem
In this study, we aim to achieve a good trade-off between the online VDD performance and the offline adaptation speed. To this end, it is necessary to address the following issues. (1) The appearance looks similar between the burr area and the background area in the image (Fig. 1). (2) The shapes and sizes of burrs are diverse and not easy to generalize. (3) The burrs occur in non-convex cylindrical holes inside the joint parts ( Fig. 2), and are thus affected by the diffused light reflection that is difficult to model. (4) Even when a calibrated eye-in-hand is used, the viewpoint can shift randomly up to about 3 pixels in terms of the image coordinate. To solve the above issues, a highly versatile and accurate machine vision method is needed.

Proposed Approach
The proposed approach consists of two distinctive stages: the offline-adaptation and online-detection stages. The adaptation stage is responsible for adapting the deep neural network to a new domain, and it is the pipeline consisting of semi-automatic annotation (3.1), model-based coordinate-transformation (3.2), tailor-made network-architecture search (3.3), and network-parameter fine-tuning (3.4). The detection stage is responsible for detecting burrs inside a given image, and it consists of the coordinatetransformation (3.2) and visual burr detection (3.5). Hereinafter, each process will be described in detail.

Semi-automatic Annotation
Annotation cost is a major part of the total cost required to adapt a VDD software to small-lot metal projects, such as HPPPs. In our case study's factory site, the annotation is provided by skilled workers in the form of bounding polygons, by using the LabelMe tool [14] (Fig. 2). As can be seen from Fig. 1, it would be difficult for a non-skilled person even to visually distinguish burrs from the background textures. Surprisingly, skilled human workers often become able to distinguish burrs with 100% accuracy after sufficient time spent training (Fig. 3). This indicates that the VDD task may not be infeasible, which has motivated us to develop an automatic VDD system.
We have been developing a user-interface for semiautomatic annotation in our factory site (Fig. 2). Using it, given a 3D CAD HPPP model and a calibrated camera, a forward/backward projection model of the camera can be obtained. These models enable transferring an annotated bounding polygon in the image coordinate of one viewpoint to that of another viewpoint. This transfer technique eliminates the need for additional annotation per different observation condition on the same HPPP type, which leads to significant reduction in the total annotation cost.

Model-based Coordinate-transformation
Majority of state-of-the-art object detection networks assume bounding box-shaped annotation [15,16,17]. In contrast, the ground-truth (GT) of burr regions in an input image often have a crescent-like shape and do not fit well into bounding boxes. To solve this, we propose to transform the image coordinate system appropriately. In our specific case, such crescent-like burrs usually occur inside cylindrical HPPPs. Therefore, polar coordinate transformation with the cylinder center coordinates in this image as a hyper-parameter is performed (Fig. 3). Through preliminary experiments, we found that such a cylinder center in a given image can be stably and accurately predicted by using a RANSAC-based circle fitting to the cylinder border circle. Figure 3 shows the result of image transformation. Comparing before and after image transformation, it can be confirmed that the filling rate of the burr region with respect to the bounding box is higher after transformation. In the experiment, polar coordinate images with size 800×1333 pixels were used.

Tailor-made Network-architecture Search
The proposed NAS framework is inspired by [18], which we further developed by introducing the following two steps, which are iterated until the time budget is expired: (1) The Controller-RNN creates a candidate architecture and trains a child network with that architecture, and (2) The Controller-RNN is updated by policy gradient with rewards obtained from reinforcement learning. The objective of the proposed NAS is to find an optimal child network, so as to maximize the VDD performance in terms of domain adaptation and generalization. In this framework, a child network is described by a parameter variable that consists of a set of mutually connected network building blocks and their connection relationship (Fig. 4). A building block takes two feature maps from the backbone network (ResNet-50) as input, and integrates them using SUM or global pooling operation, into a new feature map. Thus, each block can be described by a triplet ID: a pair of input feature map IDs and an operation ID (∈ {SUM, POOLING}). By definition, the input IDs chosen are not duplicated. In addition, the new feature map created by a block is regarded as an additional candidate input for future building blocks. Thus, the space of the triplet ID can increase as iteration proceeds. At each iteration, the newest feature map is considered as the output of a child network. In the experiments, a set of 4 feature maps P2, P3, P4 and P5, with resolutions (200, 334), (100, 167), (50, 84), and (25, 42), respectively, are used.
We observe that combining the low and high layer feature maps (i.e., combining primitive and semantic features) with NAS is often effective. The reason might be that in our application of VDD-HPPPs, the size of burrs has a large bias and the shape is not constant and thus, the high level semantic information plays an important role. The NAS efficiently searches over the exponentially large number of such combinations, and successfully finds an optimal one, as we will demonstrate in the experimental section.
The process described in Step-2 is detailed in the following. This process aims to maximize the expected reward J(θ c ) by updating the hyper-parameters θ c of the Controller-RNN by policy gradient. J(θ c ) is defined by: J(θ c ) = E P (a1:T ;θc) [R], where a 1:T is an action of Controller-RNN. Thus, it is updated by: where a 1:T is an action of Controller-RNN, m = 1 is the number of architectures that the Controller-RNN verifies in one mini-batch. T is the number of hyper-parameters to be estimated. R k is the averageprecision (AP) in the k-th architecture. b is the exponential moving average of the AP of the neural network architecture up to that point. The smoothing constant is 0.8. The learning rate is 0.1. The number of training iterations is 3000 per child network. The hyperparameters in [18] are set as follows: cfgs.LR = 0.001, cfgs.WARM-step = 750, and #trials = 500. The NAS with above setting consumes 2 weeks using a graphics processing unit (GPU) machine (NVIDIA RTX 2080). Figure 5 shows progress of NAS search.

Network-parameter Fine-tuning
The searched architecture is then used to train a burr detector. For the training, the Faster R-CNN algorithm is used. At this time, the parameters for training are set as followings. cfgs.LR=0.001. cfgs.WARM-step=2,500. Subsequent learning rates are reduced 1/10 times when the number of training sessions is 60,000, and 1/100 times when the number of training sessions is 80,000. The number of training iterations is 150,000. The time required is about 2 days using the above same GPU machine.

Visual Burr Detection
The detection process takes a query image and predicts bounding boxes of burr regions with a nonmaxima suppression. The confidence score is evaluated as the highest probability values among all the classes in the Faster R-CNN. The computation speed per image was around 3 fps.

Experiment
The proposed tailor-made VDD framework has been evaluated using real HPPP dataset in four different domains that is collected on the target factory site. This section describes the dataset, the baseline method, experimental results, and provides discussions. Figure 1 shows examples of image datasets. We collected four independent collections of images, A, B, C, and D at different domains, and manually annotated every image. The dataset size are 402, 396, 50, and 76 for A, B, C, and D, respectively. The set A is used for NAS, training and testing, while B, C and D are used only for testing.

Dataset
The set A is split into 1:2:1 subsets namely, NAS, training, and evaluation subsets. The NAS subset is used for evaluating each child network during the NAS task. The union of the NAS and training subsets is used for training. The evaluation subset is used for performance evaluation on a trained NAS-searched VDD model. As aforementioned, every training/test image is transformed to polar coordinate before being input to the training or testing procedure. For NAS and training, a left-right flipping data augmentation is applied.

Baseline Method
A deep object detector using a feature pyramid network (FPN) [19] is used as a baseline method. The FPN consists of three features, bottom-up direction, top-down direction, and potential connection, to the feature layers of different scales output by the convolutional neural network. This provides both low-resolution semantically strong features and highresolution semantically weak features. A Faster R-CNN is used for detection task. Despite the efficiency, FPN is based on a manually designed architecture, and thus, it is not optimized for a given specific application. Figure 5 shows NAS progress when subset A is used as the NAS subset. It was confirmed that the curve rises slightly as the trial proceeds, and that the Controller-RNN learned the generation of a better feature map over time through trial and error, and the search was performed adequately.

Results
As shown in the Fig. 5, the NAS score converged at 110,000-th iteration. We use the architecture at this point to evaluate the VDD performance. Specifically, the convergence is judged if change in AP values between two consecutive training sessions is equal or lower than 0.01. For performance evaluation, average precision (AP) is evaluated for different threshold values on IoU, from 0.5 to 0.95 with 0.05 increment step, and then average of these AP values is used as the performance index. Table 1 shows performance results. Let us discuss the results for trained NAS searched VDD model using the dataset A as the training set. From the evaluation result, it is confirmed that the proposed method provides better performance compared with FPN in all the test sets. Figure 6 shows example detection results for GT/FPN/Proposed (columns) for test domains A/B/C/D (rows). with bounding boxes whose confidence scores are higher than 0.5. Overall, the number of bounding boxes generated by the proposed method tended to be the same as that of the ground-truth. Exceptionally, for testing domain D, there were much false positives for both the proposed and FPN methods. This is mainly due to the dark lighting conditions. It should be noted that for FPN, the bounding boxes often do not appear, as the confidence score is often lower than 0.5.
From the above results, it could be concluded that VDD performance was significantly improved by the proposed taylor-made VDD despite the fact that the adaptation process is highly automated and efficient.

Conclusion
In this study, we presented a tailor-made visual defect detection framework that can be adapted to various domains. To the best of our knowledge, we are the first to formulate the VDD-HPPPs as an important and challenging new machine-vision application that is characterized by a combination of non-convex metal parts with complex specular reflections and highmix low-volume production. In this study, we adapted the NAS technique to search for the optimal architecture of the network. Through a factory case study, we demonstrated that our approach is able to search for a versatile network architecture and enables us to detect burrs with higher accuracy compared with the baseline method under domain shifts.


Property-driven Training: All You (N)Ever Wanted to Know About

Neural networks are known for their ability to detect general patterns in noisy data. This makes them a popular tool for perception components in complex AI systems. Paradoxically, they are also known for being vulnerable to adversarial attacks. In response, various methods such as adversarial training, data-augmentation and Lipschitz robustness training have been proposed as means of improving their robustness. However, as this paper explores, these training methods each optimise for a different definition of robustness. We perform an in-depth comparison of these different definitions, including their relationship, assumptions, interpretability and verifiability after training. We also look at constraint-driven training, a general approach designed to encode arbitrary constraints, and show that not all of these definitions are directly encodable. Finally we perform experiments to compare the applicability and efficacy of the training methods at ensuring the network obeys these different definitions. These results highlight that even the encoding of such a simple piece of knowledge such as robustness in neural network training is fraught with difficult choices and pitfalls.

Introduction
From their inception in the first half of the 20th century, to their rise in computer vision and signal processing applications in the 90s, neural networks have made their way into modern AI applications, mostly as components responsible for perception tasks, e.g. processing signals, images, speech or text.
Unfortunately, neural networks lack properties known as explainability and interpretability: we usually deploy them in domains where the ideal data distribution is unknown or unattainable. Therefore, by definition, they are only approximately correct, only as good as the data is, and do not carry additional semantic meaning beyond a very general type R n → R m . In the absence of a clear specification, it becomes problematic to guarantee the absence of "bugs", or identify components that are responsible for deviations from the intended behaviour. The most famous instance of this problem is proving robustness against adversarial attacks (Szegedy et al. 2014;Goodfellow, Shlens, and Szegedy 2015). Adversarial robustness is a property that even very accurate neural networks fail to satisfy.
As safety and security are critical for some complex AI systems involving neural networks, their verification has be-come a hot topic, with several competing tools and methods on the market: ERAN (Gehr et al. 2018), Marabou (Katz et al. 2019), nnenum (Bak et al. 2020) to name a few. However, verification tools do not entirely solve the problem as they do not directly address the issue of an under-defined specification during training. To put it plainly, there is no reason to run a computationally expensive verifier if the neural network is not robust in the first place.
The proposed solution is to (re)train neural networks towards specified robustness properties. Such training can be seen as a way to impose a formal specification, and so may contribute to explainability as well as verification. This work considers four of the most prominent families of techniques: 1. Data augmentation (Shorten and Khoshgoftaar 2019) 2. Adversarial training (Goodfellow, Shlens, and Szegedy 2015;Madry et al. 2018) 3. Lipschitz robustness training (Anil, Lucas, and Grosse 2019;Pauli et al. 2021) 4. Training with logical constraints (Xu et al. 2018;Fischer et al. 2019) The last, training with logical constraints, is a more general approach that can train for not just robustness, but a wide range of properties expressed in variants of first-order logic.
Contributions In this work, we consider all of these approaches holistically under the prism of knowledge representation. We take classification problems as an example domain, interpreting f : R n → R m as a procedure that separates the n-dimensional data into m classes. We discuss how, although the first three families of methods all seek to represent the same high-level knowledge in the neural network (namely that the true function being modelled by the network is robust to small changes in the input), underneath they optimise for different definitions of robustness. We formally distinguish standard (SR), classification (CR), Lipschitz (LR), and approximate classification (ACR) robustness as key properties that encompass all four groups of methods. This enables the following findings: • From the security perspective, different definitions of robustness ultimately determine the nature of attack, thus giving rise to SR, CR, LR, ACR attacks. Thus, one can train for example with ACR constraint but attack with SR constraint. This raises questions about relative strengths of these different training methods and attacks. • Some property-driven training methods are special cases of others. For example, adversarial training known in the literature can be seen as a form of training with SR constraint, with certain amount of parameter tuning. But some training regimes do not yield generalisations. For example, the effect of data augmentation on training cannot be emulated by training with constraints. • We can order some robustness properties based on their strength, for example, we show that LR implies SR, and ACR implies CR. In this case, training with a stronger property (e.g. LR) will protect better against both kinds of attacks (in this case, both SR and LR attacks). • There are some properties that cannot be ordered by strength (e.g. SR and ACR, LR and ACR), and in this case, optimising training for a given property defends better only against attacks with this same target property. • Training with constraints defends against adversarial attacks better than data augmentation for any choice of robustness definition as a training constraint. • There are additional common criteria that can be used to qualitatively compare different modes of property-driven training, e.g. interpretability, global or local nature. For example. CR is the most interpretable, but not globally desirable, LR is least interpretable, but globally desirable.
These are the first results of this kind. Note that some of the previous work reported on unstable performance of constraint-driven training when it comes to defence against attacks (Ayers et al. 2020), which we do not observe in our experiments. No prior comparison between constraintdriven training and data augmentation existed. Some papers, such as (Fischer et al. 2019), listed and even implemented some kinds of robustness constraints that we study here, but gave no indication of their relative performance. We are not aware of any prior analysis of SR, LR, CR, ACR abstractly as logical properties. We are confident that this general method of initial abstract property analysis followed by subsequent application to training and evaluation of networks will be of broader use beyond just classification algorithms or just robustness properties.
To our knowledge, this is also the first systematic study of property-driven training from a general "knowledge representation" point of view. Although robustness was chosen as the property of interest in this paper, we aim to convincingly illustrate that training using logical constraints generally requires careful formal treatment, as many misconceptions and even errors arise on the boundary between logical constraints (such as the definitions of robustness expressed in first-order logic) and their machine-learning realisations (given by explicit or implicit translation into the language of loss functions and statistical optimisation). At the same time, this paper aims to convince the reader that conceptual clarity in this domain is possible; and thus aims to open new possibilities for a future community project on formally expressing and analysing other properties of machine-learning algorithms. The paper is organised as follows. Section 2 explains how different robustness properties arise from different machine learning approaches to property-driven training. Section 3 abstractly analysises these robustness definitions, establishing their relative strength, interpretability and applicability. Section 4 shows how these robustness properties determine different evaluation metrics and attacks, and provides a comprehensive empirical evaluation of the robustness properties deployed as training constraints and as attacks. Section 5 concludes the paper and outlines future work.

Existing training techniques
We will explain our main ideas by means of an example. Suppose we are given the MNIST data set (LeCun, Cortes, and Burges 2010) which contains 60, 000 images of handwritten digits 28 × 28 pixels each, and a trained neural network f MNIST : R 784 → R 10 , where 784 is the number of pixels in every input image, and 10 is the number of output classes from 0 − 9. We will assume for the sake of the argument, that f MNIST is an arbitrary network trained using the standard settings of the gradient descent algorithm with some standard loss function L : R 784 → R 10 → R, that indicates the magnitude of the network's error given a data point and its intended classification. The most common example is the cross entropy loss function. We assume also that f MNIST has a high accuracy on the training set (say 99%).
For the property that we want to ensure, let us take the -ball robustness widely used in the literature (Singh et al. 2019;Huang et al. 2017;Katz et al. 2019). Intuitively, an image is robust if, when you move no more than away from it in the input space, the output does not change much, or alternatively, the classification decision that the network gives does not change. To verify that the network is robust for the MNIST data set, we check that the network is robust in the vicinity of each of the given 60, 000 images.
Take the example image in Figure 1 which is labelled in the data set as a 7. The network being robust around that image implies that any small perturbation of the image should result in only a small perturbation of the output and hence the perturbation should also classify as 7.
The problem is that for an arbitrary f MNIST , even when the is reasonable, the odds of f MNIST satisfying this con-straint for every image in the data set are extremely low. For example, the network with 99% accuracy used in (Fischer et al. 2019) did not satisfy the -ball constraint around a single image in the data set.
Let us now look at the existing solutions to this problem from the knowledge representation point of view.

Data augmentation
Data augmentation is one of the earliest and simplest methods of improving the robustness of a neural network via training (Shorten and Khoshgoftaar 2019). It is applicable to any transformation of the input (e.g. addition of noise, translation, rotation, scaling) that leaves it semantically unchanged. To make the network robust against such a transformation, one augments the data set with instances sampled via the transformation. For example, to improve the robustness of the network against simple noise, we can sample images from the -ball of the given image of 7, and add them to the data set with the label 7.
Although it may seem that this simple solution has nothing to do with knowledge representation, it imposes significant choices from the point of view of the property specification: c1 the choice of will reflect our assumptions about admissible range of perturbations; c2 the choice of the distance function that measures theproximity will reflect our assumptions on desirable geometric properties of the perturbations; c3 the choice of the sampling method (e.g. random sampling, or based on some attack, generative algorithm or prior knowledge of images) will ultimately determine the properties we optimise for. But, perhaps even more significantly for us, this method determines the exact definition of robustness that we optimise for when we train our neural network f : R n → R m . We call it classification robustness and formally define as follows: given a model inputx, for all inputs x within the -ball distance ofx, ensure that outputs f (x) yield arg max(f (x)) = c, where c is the class ofx as given in the training data. In other words: Definition 1 (Classification robustness) Used as a spec for training, this property does not account for possibility of having "uncertain" images in the data set, for which a small perturbation ideally should change the class. For data sets that contain a significant number of such images, it will lead to significant reduction in accuracy of the trained neural networks; and, as we show later it may even have a detrimental effect on a network's robustness.

Adversarial training
Adversarial training is the current state-of the-art method to robustify a neural network. Whereas standard training tries to minimise loss between the predicted value, f (x), and the true value, y, of each entry (x, y) in the training data set, adversarial training minimises the loss with respect to the the worst-case perturbation of each sample in the training data set. It therefore replaces the standard training objective L(x, y) with: max Algorithmic solutions to the maximisation problem that find the worst-case perturbation has been the subject of several papers. The earliest suggestion was the Fast Gradient Sign Method (FGSM) algorithm introduced by Goodfellow et al. (Goodfellow, Shlens, and Szegedy 2015): FGSM(x) = proj(x + · sign(∇ x L(x, y))) However, modern adversarial training methods usual rely on some variant of the Projected Gradient Descent (PGD) algorithm (Gu and Rigazio 2014) which iterates FGSM some number of times: ) It has been empirically observed that neural networks trained using this family of methods exhibit greater robustness at the expense of an increased generalization error Madry et al. 2018;Zhang et al. 2019), which is frequently referred to as the accuracy-robustness tradeoff for neural networks (although this effect has been observed to disappear as the size of the training dataset grows (Raghunathan et al. 2019)).
In logical terms what is the property that this procedure is trying to train for? Obviously it's unreasonable to expect that adversarial training will ever succeed in driving the loss of all perturbations down to zero. Therefore let us assume that there's some maximum distance, δ, that it is acceptable for the output to be perturbed given the size of perturbations in the input. This leads us to the desired property: Definition 2 (Standard robustness) SR( , δ) ∀x : ||x −x|| ≤ ⇒ ||f (x) − y|| ≤ δ where in this case of adversarial training the distance between the outputs is measured using L(x, y).
We note that, just as with data augmentation, choices c1 -c3 are still there to be made, although the sampling methods are usually given by special-purpose FGSM/PGD heuristics based on computing the loss function gradients.

Training for Lipschitz robustness
More recently, a third competing definition of robustness has been proposed: Lipschitz robustness (Balan, Singh, and Zou 2018). Inspired by the well-established concept of Lipschitz continuity, Lipschitz robustness asserts that the distance between the original output and the perturbed output is at most a constant, L, times the change in the distance between the inputs.
As will be discussed in Section 3, this is a stronger requirement than standard robustness. Techniques for training for Lipschitz robustness include formulating it as a semidefinite programming optimisation problem (Pauli et al. 2021) or including a projection step that restricts the weight matrices to those with suitable Lipschitz constants (Gouk et al. 2021).

Training with logical constraints
Logically, this discussion leads one to ask whether a more general approach to property formulation may exist, and several attempts in the literature addressed this research question (Xu et al. 2018;Fischer et al. 2019), by proposing methods that can translate a first-order logical formula C into a constraint loss function L C . The loss function penalises the network when outputs do not satisfy a given Boolean property, and universal quantification is handled by a choice of sampling method. Our standard loss function L is substituted with: where α and β are weights that control how much a standard and the constraint loss contribute to training. This method looks deceivingly as a generalisation on the previous approaches. However, even given suitable choices for c1 -c3, the robustness definition (1) cannot be modelled via the constraint loss. For (1), it work as argmax is not differentiable. Instead (Fischer et al. 2019) define an alternative property we will call approximate classification robustness: Definition 4 (Approximate classification robustness) which only looks only at the prediction of the true class and checks whether it is greater than some value η (chosen to be 0.52 in their work).

Summary
We have just shown how non-trivial knowledge representation choices and problems arise on the boundary between logical form of the desired properties and their machinelearning realisations as loss functions. In the next section, we give analysis and explanation of the rationale for making a choice between these different kinds of robustness.

Comparison of definitions
In Section 2 we showed that there were a variety of different robustness definitions used by common training techniques. In this section we perform an in-depth comparison of the definitions, a summary of which is shown in Table 1.

Relationships
One of the most important aspects to discuss is the logical relationship between the various definitions, i.e. when the definitions agree and when they disagree.
Standard and Lipschitz robustness The easiest relationship to quantify is the one between standard robustness and Lipschitz robustness. In particular, the latter is a strictly stronger property than the former, in the sense that when a network satisfies LR( , L) then it also satisfies SR( , L). However, the converse does not hold, as standard robustness does not relate the distances between the inputs and the outputs. Consequently, there is no value of L such that if the network is SR( , δ) robust then the network is LR( , L) robust, as for any fixed L one can always make the distance ||x −x|| arbitrarily small in order to violate the Lipschitz inequality.
(Approximate) classification robustness The next relationship we study is between classification robustness and approximate classification robustness. As discussed in Section 2.4, the latter is designed to emulate the former whilst providing a logical loss function with a meaningful gradient. We work under the assumption that the last layer of the classification network is a softmax layer and therefore the output forms a probability distribution. When η > 0.5 then, as would be hoped, any network that satisfies ACR( , η) also satisfies CR( ). For η ≤ 0.5 this relationship breaks down as the true class may be assigned a probability greater than η but may still not be the class with the highest probability. We therefore recommended that one only uses value of η > 0.5 when using approximate classification robustness. For example η = 0.52 in (Fischer et al. 2019).
Given that LR is stronger than SR and ACR is stronger than CR, the obvious question is whether there is a relationship between these two groups? In short, the answer to this question is no. In particular, although the two sets of definitions agree whether a network is robust around images with high-confidence, they disagree over whether a network is robust around images with low confidence. We illustrate this with an example, comparing SR against CR. We note that similar analysis holds for any pairing from the two groups.

Standard vs classification robustness
The key insight is that standard robustness bounds the drop in confidence that a neural network can exhibit after a perturbation, whereas classification robustness does not. Figure 2 shows two hypothetical images from the MNIST dataset. Our network predicts that Figure 2a has an 85% chance of being a 7. Now consider adding a small perturbation to the image and consider two different scenarios. In the first scenario the output of the network for class 7 decreases from 85% to 83% and therefore the classification stays in the same. In the second scenario the the output of the network for class 7 decreases from 85% to 45%, and results in the classification changing from 7 to 9. When considering the two definitions, a small change in the output leads to no change in the classification and a large change in the output leads to a change in classification and so robustness and classification robustness both agree with each other.
However now consider Figure 2b with relatively high uncertainty. In this case our network is (correctly) less sure about the image, only very narrowly deciding that it's a 7. Again consider adding a small perturbation to it. In the first  scenario the prediction of the network changes dramatically with the probability of it being a 7 increasing from 51% to 91% but leaves the classification unchanged as 7. In the second scenario the output of the network only changes very slightly, decreasing from 51% to 49% flipping the classification from 7 to 9. Now, the definitions of robustness and classification robustness disagree. In the first case, adding a small amount of noise has erroneously massively increased the network's confidence and therefore the robustness definition correctly identifies that this is a problem. In contrast classification robustness has no problem with this massive increase in confidence as the classification remains unchanged.
Given this example, it is hopefully clear that although robustness and classification robustness agree on lowuncertainty examples, classification robustness breaks down and starts giving what we argue are both false positives and false negatives when considering examples with highuncertainty.

Data set assumptions
A related question that we have found to be rarely discussed in the literature is what assumptions the different definitions are making about the distribution of the training data with respect to the data manifold of the true distribution of inputs.
For SR and LR it is, at minimum, desirable for the network to be robust over the entire data manifold. In the most domains the shape of the manifold is unknown and therefore it is necessary to approximate it by taking the union of the balls around the inputs in the training dataset. We are not particularly interested about whether the network is robust in regions of the input space that lie off the data manifold, but there is no problem if the network is robust in these regions. Therefore these definitions make no assumptions about the distribution of the training data set. This is in contrast to (A)CR. As discussed in the previous section, rather than requiring that there is only a small change in the output, they require that there is no change to the classification. This is only a desirable property when the region being considered does not contain a decision boundary. Consequently when one is training for some form of classification robustness, one is implicitly making the assumption that the training data points lie away from any decision boundaries within the manifold.
In practice, most datasets for classification problems assign a single label instead of an entire probability distribution to each input point, and so this assumption is usually valid. However, we feel it is important to note that classification robustness is not an appropriate definition to train for if the data set contains input points that may lie close to the decision boundaries.

Interpretability
One of the key selling points of training with logical constraints is that, by ensuring that the network obeys understandable properties, it improves the explainability of the neural network. Each of the robustness constraints encode that "small changes to the input only result in small changes to the output", but the interpretability of each definition also matters.
All of the definitions share the parameter which measures how much of a perturbation of the input is acceptable which is relatively interpretable. Despite its other drawbacks discussed so far, CR is inherently the most interpretable as it has no second parameter. In contrast, SR and ACR require extra parameters, δ and η respectively, which measures the allowable deviation in the output. While these second parameters still have a direct interpretation, their addition make these models less interpretable.
Finally we argue that, although LR is the most desirable property, it is also the least interpretable. Its second parameter L measures the allowable change in the output as a proportion of the allowable change in the input. It therefore requires us to not only have an interpretation of distance for both the input and output spaces, but to be able to relate them. In most domains, this relationship simply doesn't exist. Consider the MNIST dataset, both the commonly used notion of pixel-wise distance used in the input set, although crude, and the distance between the output distributions are both interpretable. However, the relationship between them is not. For example, what does allowing the distance between the output probability distributions being no more than twice the distance between the images actually mean? This therefore reflects the trade-off between complexity of the property and its interpretability.
Another aspect of interpretability that appears in logicalconstraint driven training, is the semantics of the parameters α and β which allow the users to decide how to weight the accuracy of the network vs the importance that it adheres to the constraint. Section 1 describes how the loss function used in logical-constraint training is split into two parts: where the first part is designed to maximise the accuracy of the predictions and the second encodes the constraint. However, in practice the properties (and therefore the derived loss functions) refer to the true label y rather than the current output of the network f (x), e.g.
This leads to scenarios where a network that is robust aroundx but gives the wrong prediction, being penalised by L C which on paper is designed to maximise robustness. Essentially L C is trying to maximise both accuracy and constraint adherence concurrently. Instead, we argue that to preserve the intended semantics of α and β it is important to instead compare against the current output of the network e.g. ∀x Of course, this doesn't work for ACR because in order to derive the most popular class from the output f (x) you need arg max, the very function ACR seeks to avoid using. This is yet another argument why using (A)CR should be avoided where possible.

Distance functions and sampling
The one aspect that all the definitions of robustness have in common is the universal quantification over the set of perturbations that are "near" to the points in the dataset. Complete procedures that guarantee to find a perturbation that violates the robustness property if such a point exists, are too expensive to run during training. Consequently all the training algorithms we consider, translate this quantifier into a sampling problem over the domain. Given this commonality, one might assume that the sampling algorithms used would be similarly homogeneous across the different training methods. However, somewhat surprisingly, this does not appear to be the case. Adversarial training algorithms in the literature typically use sophisticated gradient-based methods like PGD. In contrast, the approach to sampling in data augmentation is much more varied. Some data augmentation practices do use guided attacks such as PGD after having partially trained the network on the original dataset, but they also frequently use much less guided sampling procedures which instead seek to generate samples that are representative of the targeted perturbation space rather than actively seek adversarial examples. Common such sampling procedures include adding uniform random noise and or transforming the images via rotations/scaling/translations/palette swaps. This leads us to hypothesise that there is significant potential of crosspollination between the different training techniques.
The space in which the sampling procedure operates is inherently determined by the choice of distance function. Although L p norms are commonly used for simplicity, it is common knowledge that they are not a good proxy for semantic similarity, so restricting the sampling domain according to such a norm can introduce undesirable invariances in the neural network. It is an active field of work coming up with sample spaces such that contain more realistic sets of perturbations.
However, another important consideration when choosing a distance function is verifiability, which often discourages the use of explicitly semantic-based distance functions. In general, the more complex the notion of distance, the more difficult it is to formally verify the conditions. At one end of the spectrum the L 1 or L ∞ norms can be verified by complete verifiers such as Marabou (Katz et al. 2019). Switching to the slightly more natural L 2 norm, or Euclidean distance, means that complete verification procedures are in general unable to verify the property, and instead one is forced to rely on incomplete verifiers such as ERAN (Gehr et al. 2018). At the far end of the spectrum, very few tools are capable of verifying robustness against even comparatively simple geometric transformations such as rotations, although some recent progress has been made (Singh et al. 2019).

Experiments
In this section we describe the results of some experiments designed to validate the conclusions from our previous sections and to compare the relative effectiveness of training with logical constraints against the other training techniques. However, before describing the experiment set up, we explain different choices for measuring success of propertydriven training. This is in line with our bigger purpose of setting up a comprehensive framework for different propertydriven approaches.

Evaluation metrics
Given a particular definition of robustness, a natural question is how to quantify how close a given network is to satisfying it? We have found the terminology in the literature both confusing and inconsistent, and therefore the following discussion is an attempt to clarify matters. We note that although we are only evaluating robustness properties, the definitions below easily generalise to any property that universally quantify over some set of inputs.
We argue that there are three different measures that one should be interested in: 1. Does the property hold? This is a binary measure and the answer is either true or false.
2. If the property doesn't hold, how easy is it for an attacker to find a violation?
3. If the property doesn't hold, how often does the average user encounter a violation?
Based off of these measures, we now define three concrete metrics: constraint satisfaction, constraint security, constraint accuracy. Let X be the training dataset, and B(x, ) {x ∈ R n | ||x −x|| ≤ } Figure 3: Experiments that show how the two networks trained with LR and SR constraints, respectively (with = 0.1, δ = 10, L = 10 and L ∞ distance) perform when evaluated against different definitions of robustness underlying the attack.
and P be the right hand side of the implication in each of the definitions of robustness. The I φ be the indicator function which is defined as follows: The constraint satisfaction metric measures the proportion of the training dataset for which the property holds.
Definition 5 (Constraint satisfaction) Unfortunately, depending on the network architecture and the property in question, the indicator function may not always be feasible to evaluate. In contrast, constraint security measures the proportion of inputs in the training dataset such that an attack A is unable to find an adversarial example for property P . Definition 6 (Constraint security) In our experiments we use the PGD attack for A, although in general any strong attack can be used. Finally constraint accuracy estimates the probability of a random user coming across a counter-example to the property. Let S(x, n) be a set of n elements randomly uniformly sampled from B(x, ). Then constraint accuracy follows: Definition 7 (Constraint accuracy) Note that there is no relationship between constraint accuracy and constraint security: an attacker may succeed in finding an adversarial example where random sampling fails and vice-versa. Our naming scheme differs from (Fischer et al. 2019) who use the term constraint accuracy (without explicitly defining it in the paper) to refer to what we term constraint security. We do not propose changing the name lightly, but the term constraint accuracy caused us significant confusion, and we hope that the reader will agree that the name constraint security is more appropriate given the use of an adversarial attack.
Note the role of sampling in this discussion and compare it to the discussion of the choice c3 in Section 2.1. Firstly, sampling procedures affect both training and evaluation of networks. But at the same time, their choice is orthogonal to choosing the verification property for which we optimise or evaluate. For example, we measure constraint security with respect to the PGD attack, and this determines the way we sample; but having made that choice still leaves us to decide which constraint, ACR, SR, LR, or other we will be measuring as we sample.
Choosing an evaluation metric. It is important to note that for all three metrics, one still has to make a choice for property P , namely SR, ACR or LR, as defined in Section 2. As constraint security always uses PGD to find input perturbations, the choice of SR, ACR and LR effectively amounts to us making a judgement what consists of an perturbation adversarial: is it a class change as defined by ACR, or is it a violation of the more nuanced metrics defined by SR and LR? Therefore, for constraint security, we will be talking about SR/ACR/LR attacks in the remainder of the section. This will be further analysed in the experiment E1 below.
For large search spaces in n dimensions, random sampling deployed in constraint accuracy fails to find the trickier adversarial examples, and usually has deceivingly high performance: we found 100% and > 98% constraint accuracy for SR and ACR, respectively. We will therefore not discuss these experiments in detail, but refer the reader to the project's GitHub page (Casadio et al. 2021).
Constraint satisfaction is different from constraint security and accuracy, in that it must evaluate properties over infinite domains rather than merely sampling from them. Verifiers such as Marabou or ERAN are built with this purpose in mind. It is out of scope for this paper to conduct experiments by this measure, and so we leave it for future work. Our preliminary experiments with Marabou confirm the generally accepted assumption that it is very hard for training tech-niques to achieve a high-score for constraint satisfaction.

Experimental setup
Data sets. We use the two benchmark datasets: 1. The FASHION MNIST (or just FASHION) dataset (Xiao, Rasul, and Vollgraf 2017) consists of 28 × 28 greyscale images of clothing items; 60,000 for training and 10,000 for testing. It has 10 classes: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot. 2. The GTSRB dataset (Stallkamp et al. 2011) contains 50, 000 images of German traffic signs. There are 40 classes and image sizes vary between 15×15 to 250×250 pixels. We use a modified version here that contains 12, 660 training images and 4, 170 test images. These images are all centre-cropped, greyscaled, 48 × 48 pixel and belong to 10 classes. Each dataset represents a vision classification problem where the goal is to identify which class a given image belongs to.
Networks. For our baseline architecture, we use two fully connected layers: the first layer uses the ReLU activation function, and the second uses the clamp function to restrict each output to the range [−100, 100]. We use a clamp function instead of the traditional softmax function because the former is compatible with the property verification tools such as Marabou whereas the latter is not. The predicted classification is then taken as the output with the maximum score. For instance, f MNIST = F 0 • F 1 , where F 1 : R 784 → R 100 and F 0 : R 100 → R 10 , α 0 = ReLU and α 1 (x) = clamp(x, −100, 100).
Loss functions. Since our experiments look at classification problems, we will use the cross-entropy loss function as our baseline loss function: For the L C component of the loss function L * , we use the constraint-to-loss function translation of (Fischer et al. 2019). In all experiments, we use the Adam optimiser (Kingma and Ba 2015) with the following learning parameters: η = 0.0001, 100 epochs, a batch size of 128.
Settings. By taking the given architecture, we vary only the loss functions. Thus our Baseline network is trained just with cross-entropy. Data Augmentation adds 2 additional images in the 0.1--ball of each image, and it samples either randomly (RU) or using an FGSM attack. Adversarial training refers to the standard PGD attack, or alternatively a L * function where β = 0, and α = 1, with PGD sampling. In all other cases, we use a constraint loss function L C defined as in (Fischer et al. 2019), and we use the constraints SR, ACR, LR as defined in Section 2 with α = 1 and β = 0.2. Because the loss function translation given in (Fischer et al. 2019) is not uniquely defined in its base case, for the sake of further control experiment, we define the constraint adversarial robustness (AR) which uses cross entropy for L C . All networks trained with L * use sampling by the PGD attack, for efficiency (as well as comparability).

Results
We start with noting the standard test set accuracy of the resulting neural networks in Table 2, making sure that our different training regimes do not deteriorate networks' general performance too drastically. Most notable accuracy drop is noticed for adversarial training. We now highlight groups of experiments that confirm or extend our main theoretical conclusions; the complete experiment description is available in (Casadio et al. 2021).
Experiment set E1. Comparable properties. In Section 3, we established that LR as a property is stronger than SR, and both are not strictly comparable to ACR. This would suggest that, if we train two neural networks, one with the SR, and the other with the LR constraint, then the latter should always have higher constraint security against both SR and LR attacks than the former. This is indeed confirmed by the experiment shown in Figure 3. We also discussed that depending on the properties of the data set, SR may not guarantee ACR, and Figure 3 shows exactly that case. It also confirms that generally, stronger properties are harder to obtain: whether a network is trained with SR or LR constraint, it is less robust for LR attack than any other attack.
Experiment set E2. Incomparable properties. The results in Figure 4 tell us that using the ACR constraint for training does not help to increase defences against SR attacks. A similar picture, but in reverse, can be seen when we optimise for SR but attack with ACR, see (Casadio et al. 2021).
Experiment set E3. Constraint training versus other modes of property-driven training. Next, we confirm our assumptions about relative inefficiency of using data augmentation compared to adversarial training or training with constraints, see Figure 5. We show this only for SR attack, but graphs for SL and ACR attacks show the same trends, see (Casadio et al. 2021). Surprisingly, neural networks trained with data augmentation give worse results than even the baseline network. The fact that Constraint Loss (SR) generally performs better than Constraint Loss (AR) shows that the custom-made loss function of (Fischer et al. 2019) has comparable performance to cross entropy. It is also encouraging to see that Constraint Loss (SR) is almost as good as standard adversarial training while the latter also gives 11 percentage point drop in terms of the networks accuracy (Table 2).  Experiment set E4. Role of other parameters. As already discussed, random uniform sampling struggles to find adversarial inputs in large searching spaces. It is logical to expect that using random uniform sampling when training will be less successful than training with sampling that uses FGSM or PGD as heuristics. Indeed, Figure 5 shows this effect for data augmentation, but similar trends are expected for any form of training.
Finally, one may ask whether the trends just described would be replicated for more complex architectures of neural networks. In particular, data augmentation is known to require larger networks. By replicating the results of Figure 5 with a large 18 layer convolutional network of (Fischer et al. 2019) indeed confirms that larger networks handle data augmentation better, and data augmentation improves robustness compared to the baseline. Nevertheless, data augmentation still lags behind all other modes of property driven training, and thus this major trend remains stable across network architectures, see (Casadio et al. 2021).

Conclusions, Related and Future Work
We have presented a comprehensive study of propertydriven training from the knowledge representation point of view. Taking robustness as a representative property, we ab-stractly studied different forms of robustness; and showed how the existing literature on property-driven training can be understood through this prism. Moreover, we proposed a method that separates out the logical study of properties from their implementation as loss functions for training on the one hand, and their use as evaluation methods and attacks on the other. We showed that this method allows to make general conclusions about relations of different modes of property-driven training that were not possible before.
For translation of constraints into loss functions, we used the implementation from (Fischer et al. 2019); thus our results are compatible and comparable with that prior study. In particular we have identified that the method of (Fischer et al. 2019) is not a strict generalisation of the other techniques and that significant trade-offs and decisions have to be made in order to represent some of the robustness definitions.
The main line of our future work is to extend our methods and conclusions to verification (or constraint satisfaction) frameworks such as Marabou (Katz et al. 2019) or DeepPoly (Singh et al. 2019); and obtain more flexible and general programming interaces for continuous verification (Komendantskaya, Kokke, and Kienitz 2020) that incorporates neural network verification and training.


s2fieldsofstudy:
Computer Science
Mathematics

text:
Computational Cost Reduction of Transaction Signing in Blockchain

Nowadays, Blockchain is a disruptive technology, particularly in the financial context. Moreover, Blockchain is behind the success of cryptocurrencies, e.g., Bitcoin and Ethereum. Unlike traditional currencies, cryptocurrencies are entirely virtual. There is no physical money, but it can directly make payments in digital currency from one person to another without intermediaries. Moreover, Hashing's cryptographic algorithm makes Blockchain resist tampering from any transacting participants because the submitted block cannot be altered or re-engineered. However, another big problem is how users of cryptocurrencies stop somebody from adding or editing a transaction that spends someone else's money to them. To do this, Blockchain needs another cryptosystem called Public/Private Keys, a primitive asymmetric cryptosystem, e.g., the RSA encryption, to sign the transactions for proving the authenticity of the ownership without revealing the signed secret information. The generated public key is regarded as a ledger account number or digital wallet of the sender and the recipient. Simultaneously, the paired private keys are used to identify whether the digital wallets' owners are authentic. As growing network entities and propagated Blockchain transactions, computing millions of replicated tokens in the blocks to sign and verify the digital wallet's ownership is computationally expensive. However, a certain of chosen arithmetical transformations that can simplify mathematical cost can significantly reduce computational complexity. This research's main contribution is developing a protocol that can reduce the complexity and mathematical cost in generating the digital wallet and verifying its authenticity of ownership. Finally, performance analyses of the RSA algorithm for the protocol have been measured and visualized

Introduction
Nowadays, Blockchain is a disruptive technology. It is a shared database storage technology or known as Distributed Ledger Technology (DLT). Replicated data logging guarantees that the previously recorded data is safe, seeing the same data from all network users and cannot be converted or edited. Such a mechanism provides the reliability of shared data using cryptography principles and the ability of distributed computing. The start of Blockchain technology firstly took place in 2008 with Satoshi Nakamoto's presentation [1]. It is a conceptual expression of creating a secure platform that can create security in exchanging digital currency named Bitcoin. It is unnecessary for intermediaries, such as banks or other entities involved in the payment process. It receives widespread attention and acceptance from experts worldwide that it is technologically advanced and capable of using financial and banking aspects and other major sectors, including the government, science, and education sectors. The Blockchain technology links all information together throughout the system. When a new transaction occurs, it must be announced to every machine in the system to recognize it. The transaction must also receive a consensus from the entire network before the transaction can be recorded to the Block. Therefore, Blockchain technology does not require an intermediary to do the job of organizing it. All the information is stored under the Blockchain structure and distributed to the local affiliate members. If someone tries to create a fake transaction, the data will conflict with the other members' device's data since the device connects to all other data, leading to tolerance to create such a transaction to the system. There will only be transactions that everyone in the affiliate accepts that can be recorded. The registered transactions that have been logged into the Blockchain system cannot be changed or modified. It is the main reason that makes Blockchain technology widely accepted as technology storage with high reliability. The components of Blockchain consist of four essential elements: 1) Block 2) Chain 3) Consensus and 4) Validation. Blockchain data entry is calculated in Block type layout, and each Block links to the previous Block with Block's Hash value. They are always cascaded together into chains, making it difficult to counterfeit. They can check the data blocks' integrity throughout the Chain, which can be traced back through the starting Block called Genesis Block. Block is a data packet divided into two parts. The first one is a Block Header part of letting us know what is inside the data box. The second one is a Block Data part to contain various information, such as information on the amount of money, transfer information, medical history information, and other miscellaneous information. Chain is the principle of memorizing every transaction of everyone in the system, recording the data, copying the ledger, and distributing the ledger account to its people. The system will distribute a copy of the ledger account to everyone in the network to be aware of the conflict transactions. Since the launch of the Blockchain system, www.psychologyandeducation.net even if one node is corrupted, it can confirm or re-enter the other node's valid ledger, contributing to the system's availability. Consensus is an establishment of an agreement and mutual commitment between members of the Blockchain affiliate. All of the members must agree to the rules and a mechanism to control the data's accuracy in all nodes through different algorithms to ensure the same precision and integrity, including information. There are several Consensus processes, such as Proof-of-Work, Proof-of-Stake, Practical Byzantine Fault Tolerance (PBFT), and Proof-of-Authority. Validation is an entire-system proof of validity and accuracy of data to ensure that no information is missing. The part of the Consensus is Proof-of-Work. Proof-of-Work is creating a Consensus using solving complex mathematic problems that takes a long time from nodes belonging to the network, known as Miners. The goal of Proof-of-Work is to verify the reliability of the information being logged into the network. Miner will be rewarded for doing Proof-of-Work. Consequently, the Block will gradually be chained and is always available without any system administrator. Bitcoin is an excellent example of making a Consensus using a Proof-of-Work, which is a public Blockchain. Although Blockchain technology is novel and cutting-edge, it is from nothing more than ancient knowledge called cryptography. There are two main cryptosystems used in Blockchain. The first one is Hashing, and the second one is Public/Private Keys. One of the most spread-used algorithms in Blockchain is Hashing. It is for creating a digital fingerprint of the datathe Hash changes if and only if the data changes. Hashing is also used for Mining. According to the specified conditions, the miner who can first find the suitable Hash with the corresponding "number only used once" (Nonce) will be rewarded. The designation that every node has to find the Nonce by solving a mathematical problem, known as Proofof-Work, where the accepted Block must answer the Block's aforementioned mathematical problems. Therefore, a malicious attacker cannot create a fake transaction without modifying Nonce's mathematical answer, including Block and all other Chain's blocks. Moreover, they must be done simultaneously in every Block to be accepted by other nodes, called the Consensus. That is difficult and almost impossible with the ability of today's computers. Consequently, Blockchain is regarded as a form of recording highly secure data. Another cryptographic primitive used in Blockchain is Public/Private Keys. It can be used not only for creating a digital wallet but signing transactions as well. A keypair generator creates digital keys. The public key is the address of the wallet, which is approximately 512 character-long hexadecimal characters. These numbers are used as a bank account number of the sender and the recipient. Simultaneously, the paired Private key is used to sign and verify the ownership of the wallet. The private key must be first generated using random numbers, but it needs to be kept secret as the name implies. Only the authentic wallet's owner knows the private key. After that, the public key is generated by using the chosen private key encrypted with asymmetric algorithm like RSA. Anyone in the network can know the public key because it is regarded as the wallet number. Finally, the cryptosystem can prove knowledge of a fake transaction without revealing the private key. In other words, the cryptographic mechanism can detect whether the transaction was tempered without using the private key. Consequently, the malicious attacker cannot change the signed transaction information because even if they know the sender's and the recipient's public key, they do not know the private key to re-sign it. The digital signature will change immediately when the malicious node changes the information in the transaction. The wallet's ownership is established through digital keys, the wallet's address, and digital signature. The digital keys are not stored in the network but are instead created and stored by users in a simple file called a wallet. Most transactions require a valid digital signature in the Blockchain, which can only be generated with the owner's private key. Keys come in pairs consisting of a private (secret) key and a private key. The public key is similar to a ledger number, and the private key is similar to the secret PIN or signature on a cheque. In practice, the popular cryptocurrency like Bitcoin is structured as a peer-to-peer network topology. The practical Bitcoin network consists of between 5,000 to 8,000 listening nodes running various Bitcoin reference clients [2]. All nodes include the routing function to participate in the network and might include other supporting functionality, e.g., the Blockchain database, mining, and wallet services. All nodes provide to validate and propagate transactions and blocks-accordingly, the more extensive network and longer propagated blocks, the more transactions to be validated. A transaction consists of four pieces of information to be signed: an amount of payment, sender's account, recipient's account, and sender's private key.
Then, when the information is signed, the signed transaction (message signature) is sent to the recipient with three public information pieces. Meanwhile, the private key is kept secret with the sender. The web-based transaction signature simulation is shown in Fig. 1.  For example, if a malicious node changes the amount of payment, even a cent, the screen will turn red which means the verification failed, as shown in Fig. 3.

Fig.3 Amount of payment changed, Alice's transaction verification failed
Transactions are the most important part of the cryptocurrency system. Everything else in cryptocurrency is designed to ensure that transactions can be created, propagated on the network, validated, and finally added to the global ledger of transactions (the Blockchain). Therefore, researchers have realized the importance of bandwidth overhead spent on signing and verifying transactions in Blockchain. Researchers believe that certain arithmetic transformations can reduce the computational cost of signing and verifying transactions. In the next section, researchers will define our contributed research objectives of our academic paper.

Research Objectives
The objectives of this research article are 1) to develop a protocol that can reduce the complexity and mathematical cost in signing and verifying Blockchain transactions, and 2) to analyze performance of our protocol measured and visualized using Python.

A. Research Process
The researchers developed a web-based transaction signature simulation using Python Flask to demonstrate signing and verifying Blockchain transactions. The processes of implementation consist of 1. Measuring runtime spent on specific numbers of transactions signing and verifying based on the 2048-bit public RSA key-size represented by 512 hexadecimal characters and the private RSA key-size varied between 5 and 512 hexadecimal characters, which is the typical range of key size for Bitcoin. 2. Simplifying the complexity of computational cost using certain arithmetical transformations and measuring them again. 3. Visualizing the summarizing of the findings by Python. 4. Analyzing our protocol's performance compared to the original schemes.

B. Rules of Our Protocol
In the first step of the research process, the researchers defined the security model for creating keypairs and signing transactions, which sk A is Alice's private key, pk A is Alice's ISSN: 00333077 1499 www.psychologyandeducation.net public key, sk B is Bob's private key, pk B is Bob's public key, m is the amount of payment Alice sending to Bob, and tx A is the transaction signed by Alice as shown in Fig. 4.

Fig.4 Research process for keypair generations and signing transaction
The other security scheme is shown in Fig. 5, which is verifying transactions.

Fig.5 Research process for verifying transaction
In the second step of our research process, the RSA algorithm's modular exponents were simplified by certain arithmetical transformations. Although the formulas in each signing and verifying scheme are different, they concern with the Legendre symbol. Our protocol is implemented on the Legendre symbol and existing arithmetical transformations. We notice that the computational time of the Jacobi symbol increases exponentially when the key size increases. Therefore, our primary goal is reducing the "numerator" modulo the "denominator". The following are the rules of our algorithm. Note that the steps of each scheme are the same, but the private and public keys (the "numerator" and the "denominator") 1.
Reformulating the Legendre symbol to Euler's criterion 2.
Reducing the "numerator" modulo the "denominator" using the Extended Euclidean Algorithm 3.
Extracting any factors of 2 from the "numerator" using the second supplement to the law of quadratic reciprocity 4.
If the "numerator" is 1, it gives a result of 1. If the "numerator" and "denominator" are not coprime, it gives a result of 0 5.
Otherwise, the "numerator" and "denominator" are now odd positive coprime integers so that we can calculate the time taken The above algorithm leads to an efficient O ((log a)(log b)) [3] algorithm for calculating the Jacobi symbol, analogous to the Euclidean algorithm for finding the greatest common divisor of two number.

C. Experiment Machine and Tools
The experiments have been done on Dell Optiplex 9010 Quad Core i7-3570 3.40-3.80 GHz CPU, 16GB of RAM. The machine runs Python 3.8 on 64-bit Windows 10 Professional.

Results
This section explained the details of the implemented experiments compared with existing algorithms. The parameters considered to evaluate the protocol's performance are keys generation time, signing time, and verifying time. Each run was done ten thousand times, and the average and standard deviations are shown on the graphs. Fig. 6 is the graph of time in seconds taken for using the RSA cryptosystem against the keys' size in numbers of digits. The value and x are fixed as five digits, and N varies from 128 to 512 digits. -The green line showed the time taken without any arithmetic algorithm.
-The blue line showed the time taken using Jacobi symbol properties.
-The red line, almost on the X-axis, showed the time taken using our protocol. To close up the graph plotted in Fig. 6. The runtime by our protocol varies from approximately 50 -90 milliseconds, which is very small as shown in Fig. 7. www.psychologyandeducation.net In maximum, the numerator should be up to 512 digits. Therefore, we changed the number of digits of the numerator to 512 digits. The graph of the time taken of each algorithm is plotted in Fig. 8. The graphs of ordinary modular exponentiation significantly increased while the graph of both the Jacobi symbol algorithm and our protocol slightly increased in the time taken. However, our protocol's graph trended to be lower than the Jacobi symbol's when the numbers of digits of N increased.

Discussions
This section discussed our demonstrated protocol's performance analysis. The experimental results of the transaction signature procedures indicated that our protocol consumed the least amount of time compared to both the original modular exponential and the Jacobi symbol transformation, not only the small but also the large numbers of private keys. Moreover, it tended to be more significantly different from our scheme's slope and others' when the public key's numbers of digits increased. More importantly, our algorithm consumed at least 6 times less than the original one's at the 512-digit key-size of the public key, or approximately 83.33% reduced. Additionally, it tended to be many more times when the public key's size was growing.
As a result, our protocol can significantly reduce the computational cost for Blockchain's transaction signing and verifying.

Recommendations
The researchers have recommendations for Further Research as followings: 1. The experiment tested in this study were taken at a maximum 512-digit RSA key. The next research should be done longer key size to see how much our protocol has better performance than other schemes. 2. The experiment should be done with asymmetric-key encryption with another RSA-like algorithm, e.g., Goldwasser-Micali cryptosystem.
3. The computational cost should be reduced by another arithmetical transformation, e.g., Chinese Remainder's Theorem or Fermat's Little Theorem as well.

Conclusion
In conclusion, transaction signing and verification are regarded as the most critical part of Blockchain. The signed transaction guarantees the protection of a data modification to be safe and secure. The transactions tend to be more important as the propagation of blocks in the Blockchain contributed to the much more signing and verifying time consumption. The findings from the analyses of both small and larger key size of the private key showed that the runtime consumed by our algorithm is very compact. It is supposed to decrease much time to sign and verify transactions in practical Blockchain. Our contribution is computational cost reduction of transaction signing in Blockchain


Establishments of Empiric Intervening Formulas Methods, Procedures and Mathematical Elementary Algorithms: Medical Applications

Background: updating methods, procedures and mathematical elementary algorithms, constitute a tool of work for the reliability of the results of any study, and in medical applications. Aims: confirming the behavior of the experimental intervening data the modelation, through the linear regression equations, according to the methods, procedures and mathematical elementary algorithms, the ones that they are not necessary in knowledge and abilities of differential calculus. Method: they use the theoretic methods: Analysis synthesis, induction deduction and abstraction concretion. Process of understanding, explanation and interpretation. Methods, procedures and mathematical algorithms, as well as statistical methods and information-technology professional programs are applicable. Results: empiric formulas follow on from a mathematical model, which confirms the behavior of the experimental data by means of the simulation, a medical problem gets worked out, the correlation coefficient is checked for sampling, a confidence interval of the linear correlation coefficient estimates itself, helped in information-technology professional programs. Conclusions: they indicate methods, procedures and the mathematical elementary algorithms to construct a model, which confirm the behavior of the experimental data, and theoreticians for simulation, as from nature and the logical order of the study, helped for information-technology professional programs.

Introduction
It is important to understand, explain and interpret the experimental data, by means of a model to establish, which verifies the information, the nature and the logic of the data, according to the experimentation for the collection, processing, presentation and interpretation of the processed data, from the efficient use of mathematical and statistical tools designed for these purposes [1][2][3].
It is necessary to master the concepts and definitions, methods, procedures, and mathematical algorithms, which facilitate the handling of information for the interpretation of the scientific problem that is solved in a study. [1][2][3] Use as examples those that involve parameters and statistics known to students, such as the linear correlation of random variables, and examples carefully selected for their simplicity and importance according to the results obtained from the available scientific literature [1][2][3]. The importance of these variables, and the relationships to be established, are assumed from the components: academic, labor and research. [1] It is necessary to have professional problems, in which statistical inference techniques are used so that they value as a result making the decisions, [1][2][3] to be followed in diagnoses and therapies to be applied in individuals, families and communities, where they provide their services of Health.
In general, significant guidelines and regulations are specified and clarified for the optimal development of the training of health professionals in undergraduate and graduate studies, from the didactic and scientificmethodological point of view.
In the teaching-learning process, patterns should not be set in the way of proposing and solving different situations by way of examples, exercises, and problems to avoid stereotypes, which are transmitted among the teachers of a discipline, for years of work, based on the results obtained.
Therefore, the following scientific problem is defined: How to establish a linear regression model for medical experimental data, based on the mastery of the contents of elementary mathematics, without resorting to the classical knowledge and skills of differential calculus?
The objective of this work is to confirm the behavior of the experimental data represented, according to elementary mathematical methods, procedures and algorithms, in which the knowledge and skills of differential calculus are not necessary.

Materials and Methods
Theoretical methods are used: analysis -synthesis, induction -deduction, abstraction -concretion. The method to construct curves of elementary functions, [4][5][6] comprehension, explanation and interpretation processes. Mathematical and statistical methods, procedures and algorithms are applied.
The empirical methods applied are mainly framed in participant and non-participant observation, in order to delve into the problem and obtain information on the main difficulties that affect the teaching-learning process, from the classes, as forms of teaching organization. Socialization workshops are held to assess the viability of the effectiveness of the proposed didactic alternative.
They select 10 professors from the discipline of Health Research Methodology of the Department of Medical Informatics of the Holguín Medical Sciences University are nominated, the intended sample according to their work experience in teaching, and interviews are carried out with open questions to identify: The beliefs, opinions, and motivations of health professionals about the usefulness of inferential statistics in professional practice. The following difficulties are revealed: Variations in the way of analyzing and using the concepts studied are not always favored in the different teaching activities in undergraduate and graduate studies in order to stimulate creative thinking in medical problem situations that require it.
1. There is no consolidated work strategy to delve into alternatives that develop creative capacities of health professionals. 2. Little use of professional statistical packages to solve professional problems and ratify the results obtained. 3. The creation and elaboration of situations that break with the schemes of traditional teaching, based on the systematization and integration of undergraduate and postgraduate content, is insufficient. A documentary review of the programs of the discipline Methodology of Health Research is carried out, in general the curricula of Medical Sciences, previous and current, the programs of the Master's Degree in Primary Health Care, scientific review of books, articles, and monographs referring to inferential statistics.

Results
The objectives proposed in different programs, the methodological guidelines, stimulate the creative thinking of teachers and students, thus stimulating the development of mental capacities, given the bilateral nature of the teachinglearning process.
It is important to create health situations to be faced by the General Practitioner in his professional work that show the need to interpret the association between random variables that describe the evolution of processes in individuals, families, and communities to consolidate the knowledge and skills related to the Concepts addressed in inferential statistics, from experimental data.
The analysis of the following problems is proposed, which are previously oriented, which guarantees the active and creative participation of health professionals. [7][8][9] The empirical formulas to be established are obtained in the updated scientific literature on the application of the contents of Higher Mathematics; However, this discipline is not taught in the Medicine career, which is why it constitutes a challenge, the search for an alternative from the basic mathematical knowledge and skills to deduce these equations from the least squares method (MCM) [10][11][12]. Problem 1. Establish the empirical formulas for the points (xi; yi) located approximately on the line r, defined by the equation y i =a + b x i , the ordinate and abscissa of the points (x i ; y i ) with i=1, 2,…, N; They are obtained as results of experimental measurements that are examined as data, for the coefficients a and b of the equation of the line, which constitute unknown quantities to be determined.
The following illustrates how medical problems are solved using the MCM. [12,13] Problem 2. Given the results of the measurements of the magnitudes of doses in milligrams (mg) of a certain drug, and the reduction of the heartbeat per minute (beats/min).
a. Determine the regression equation. b. Mean the regression equation according to the study. c. What is the dose for a reduction of 13 beats/min? d. Establish the rate of change of the reduction in heartbeat and the dose of the drug in patients. e. Check the error of the experimental data and the regression equation [14,15]. It is important to check the results obtained in the study sample at the population level, for which the following is proposed: Problem 3. Verify the correlation of the results, from contrasting the following statement: The correlation coefficient is less than or equal to 0.70 at the population level with a significance level of 5%. Given the following measurements of the dose (mg) of a drug to reduce Mathematical Elementary Algorithms: Medical Applications (beats/min) of the heart in patients, based on a randomly selected sample, making a comparison [15][16][17][18][19][20].
It is not enough to confirm the results, it is revealing to specify between which values the linear correlation coefficient is found, for other samples of that population. Problem 4. Estimate a 95% confidence interval for the correlation value in the population of the Reduction (beats/min), according to the dose (mg) of the drug, proposed in problem 3.

Discussion
Problem 1: The differences of the ordinate are given by the equation To determine the coefficients a and b in such a way that the errors are in absolute value as close to zero (as small as possible); thus the sum of the squares of the errors is as small as possible, it is designated = ∑ In other words, the sum of the errors in absolute value is minimal. When substituting, the following expression is displayed: The difference of the function U, which depends on a and b, around the point (a 0 ; b 0 ) follows as: values positive reals . The difference is developed ; [9][10][11]: thus the following addends are obtained: reaches its minimum value, if the difference: the experimental data are chosen in such a way that ∑ ≥ 0 the product coefficients and the squares of are positive real values, which are eliminated to reduce the difference of the increment of the function at the point (a 0 , b 0 ), by imposing the required conditions, cancel the linear coefficients of . The following system of linear equations is obtained (9): The formulas for the coefficients and are determined. When then, the system of equations has a unique solution, which is possible because they are conditions that the researcher imposes on his experimental design, and is part of the scientific rigor, to guarantee that the information collected provides the equation of the mathematical model, which adjusts the results obtained.
Sometimes given the nature, and the behavior of the logical order of the experimental data, do not allow to impose these required conditions; In other words, there is not always a straight line that optimizes the results obtained by experimentation, that is, it does not have a square error close to zero.
The inequality + ± ; + ± − ; ≥ 0 (11) holds for any positive real values . That is, the function U (a; b) that depends on the independent variables a and b, has a %í %' ()*+, -' which is classified as an absolute extreme. Comparing to the proposal accomplished by Escalona-Fernández [6,10]. To carry out calculations and checks, it is recommended to use computer programs, such as Liber Office Calc, which provides valuable help. The experimental data are organized in tabular form, as presented in problem 2, in accordance with the formulas to be used. A file must be previously created with the records of the information to be processed.
In the worksheet, the data option is activated, then options are displayed, when pointing in the last panel of the "Statistics" list, options are offered with a double click, access to "Regression", in this the columns with the registered data x i , y i ; Thus, the values of the "Slope" and the b. It is established that the reduction in heart beats in patients is 4.09 times the applied dose, and 7.05 beats per minute are added to the reduction due to other factors not considered in the study.
c. Then, 13 beats/min =7.05 + 4.09 * (dose mg ), then dose mg =(13 -7.05) / (4.09), that is, the dose is equal to 1.45 mg of the drug for a reduction of 13 beats per minute. d. Increasing the dose of the drug applied to patients by one milligram (1 mg) reduces approximately 4 (4.09) beats per minute.
e. The experimental data are evaluated in the (theoretical) regression equation, the calculated error is expressed, this is almost zero, therefore, the efficiency of the established mathematical model is ratified, the real and theoretical results are compared by simulation.
Making a comparison [11]. An effective behavior of the drug is manifested, from different doses, in different patients, according to the study measurements.
The ratification of the regression model is related to the verification of sufficient conditions, based on the necessary conditions imposed. On the contrary, when the error is not small, the regression model: Linear, logarithmic and potential does not fit the experimental results.
Problem 3: 1) Hypothesis formulation: H 0 : ρ ≤ 0.70, the linear correlation at the population level is less than or equal to 0.70. H 1 : ρ > 0.70, the correlation is greater than 0.70.
2) The significance level is selected α=0.05 3) The T-student distribution is used for the calculated statistic: T cal =4.85 4) T 0.95; 12 GL =2.18. As a criterion statistic (the statistician) belongs to the critical domain (because T 0.95; 12GL =2.18 < T cal =4.85). Therefore the null hypothesis is rejected.
5) It is convenient to assume the correlation coefficient at the population level is greater than 0.70 at the population level with a significance level of 5% and 12 degrees of freedom.
With a confidence level of 95% and 12 degrees of freedom, the true value of the correlation ρ in the population is between the values 0.84 (lower limit); and 0.99 (upper limit), that is: 0.84 < ρ < 0.99.
In general, it is important to establish an association between random variables, whose conditions to be imposed meet the requirements demanded by the researchers, according to the design of the experiment, and the aspects to be checked, which are illustrated in the solutions to problems 1, 2, 3 and 4; These are basic considerations to be followed as an alternative to the proposed didactic strategy.
The group of teachers, from the bank of problems created to develop the topic referred to inferential statistics. It proposes a didactic strategy to analyze and use the studied concepts applied to professional problems, in order to stimulate creative thinking.
The alternatives to develop creative capacities in teachers and health professionals, as well as the problems to be elaborated must be based on health situations to be faced by the General Physician, previously oriented; the results of which are verified by professional computer programs.

Conclusions
The basic consideration of the didactic strategy that is proposed as an alternative to be followed in this work indicates: 1. Methods, procedures and elementary mathematical algorithms to build a linear regression model, which confirm the behavior of experimental and theoretical data by simulation, according to the nature and logical order of the study. 2. The empirical formulas are established by the MCM, in which the knowledge and skills of differential calculus are not necessary, these procedures; and formulas are applied to medical problems, supported by professional computer programs, the sample correlation coefficient is verified, which is estimated by an interval at the population level. It is recommended to carry out methodological and collective workshops with the professors of the discipline Methodology of Health Research to deepen the design of the didactic strategy that is proposed in order to stimulate creative thinking, and perfect the bank of problems, as well as updating methods, and statistical inference techniques.