HyCA: A Hybrid Computing Architecture for Fault Tolerant Deep Learning

Hardware faults on the regular 2-D computing array of a typical deep learning accelerator (DLA) can lead to dramatic prediction accuracy loss. Prior redundancy design approaches typically have each homogeneous redundant processing element (PE) to mitigate faulty PEs for a limited region of the 2-D computing array rather than the entire computing array to avoid the excessive hardware overhead. However, they fail to recover the computing array when the number of faulty PEs in any region exceeds the number of redundant PEs in the same region. The mismatch problem deteriorates when the fault injection rate rises and the faults are unevenly distributed. To address the problem, we propose a hybrid computing architecture (HyCA) for fault-tolerant DLAs. It has a set of dot-production processing units (DPPUs) to recompute all the operations that are mapped to the faulty PEs despite the faulty PE locations. According to our experiments, HyCA shows significantly higher reliability, scalability, and performance with less chip area penalty when compared to the conventional redundancy approaches. Moreover, by taking advantage of the flexible recomputing, HyCA can also be utilized to scan the entire 2-D computing array and detect the faulty PEs effectively at runtime.

I. INTRODUCTION
The great success of deep learning motivates the deployment of deep learning in numerous domains of applications. Many of the applications such as autonomous driving and drones [1], [2], and intelligent medical monitoring and treatment [3] are closely related to the safety of human beings and are mission-critical. When deep learning models are applied in these applications, the reliability of the execution is of vital importance and must be considered comprehensively [4], [5]. Otherwise, the unexpected inference predictions may lead to catastrophic consequences [6]. While the deep learning models are increasingly implemented on customized deep learning accelerators (DLAs) for the sake of both higher performance and energy efficiency [7], the reliability of the model execution dramatically depends on the underlying accelerators [8], [9]. At the same time, DLAs fabricated with continuously shrinking semiconductor technologies are more likely to suffer manufacture defects and become more sensitive to the working conditions such as the large temperature variation than before [10], which may cause hardware faults and incur considerable prediction accuracy loss. Thereby, resilient DLAs are indispensable for reliable inference and are highly demanded by the mission-critical AI applications [11].
DLAs typically consist of a large regular computing array which can either be a systolic array or a plain mesh array [12], [13], and a set of on-chip buffers used for input features, output features, and weights. While the on-chip buffers implemented with SRAMs can be usually protected effectively with error correction code (ECC), we mainly focus on the reliability design of the regular computing array in this work. As each processing element (PE) in the computing array can be used for the calculation of multiple features in different network layers, faults in a single PE may cause multiple faulty computing results during the deep learning model execution. Thereby, they may result in considerable accuracy degradation according to our experiments in Section III.
To mitigate the hardware faults in the 2-D computing array of DLAs, researchers have proposed a number of faulttolerant design approaches from distinct angles. These approaches can be roughly divided into two categories. The first category mainly exploits the inherent fault-tolerance of the neural network models by retraining the neural network models specifically for the faulty computing array without any modification or with minor modification to the computing array [14]- [17]. Although these approaches induce negligible hardware overhead and can even be applied to many off-theshelf accelerators, the model retraining is required for each specific fault configuration, and the retraining, especially for large data sets and deep learning models, can be rather expensive. For instance, a critical neural network model applied in automotive systems must go through a series of standard certification tests before the modification can be accepted [18]. The cost of the certification test is nontrivial and timeconsuming. Moreover, the prediction accuracy of the retrained models depends on both the model structures and the specific fault configurations. There is no guarantee that the model retraining can always maintain the original model accuracy and fulfill the requirements of mission-critical AI applications arXiv:2106.04772v2 [cs.AR] 27 Oct 2021 for all the different fault configurations by design.
The second category aims to recover the faulty computing array with redundant PEs. While the conventional approaches such as dual modular redundancy (DMR) and triple modular redundancy (TMR) require substantial hardware resources, researchers proposed a variety of relaxed redundancy approaches. The basic idea is to have each redundant PE to recover any faulty PE in a limited region of the computing array while the region can be a row, a column, or both row and column [19] [20], which essentially limits the sharing of the redundant PEs and reduces the hardware resource consumption significantly compared to the DMR and TMR approaches. When the faulty PEs are evenly distributed across the computing array, the faulty PEs can be probably mitigated. Nevertheless, the faults may not be evenly distributed across the computing array and these approaches fail to recover the computing array even when the number of redundant PEs exceeds the number of the faulty PEs in the computing array. In this case, the DLAs will not be fully functional or degrade dramatically if the faulty PEs are discarded. In summary, there is still a lack of resilient computing array architectures for DLAs that allow unmodified deep learning model execution and can tolerate various fault distribution at the same time.
To address the problems, we propose a hybrid computing architecture (HyCA) for the fault-tolerant DLAs. Unlike prior works in which each homogeneous redundant PE can only protect a small subset of PEs in the 2-D computing array, we have a set of dot-production processing units (DPPUs) implemented as redundancy targeting at any faulty PEs in the 2D computing array. DPPUs can have all the output feature calculation that is mapped to the faulty PEs in the 2-D computing array recomputed. The locations of the faulty PEs in the 2-D computing array will not affect the recomputing. When the DPPU computing power is lager than the recomputing requirements, the neural network execution can be recovered without any performance penalty nor accuracy loss. When the DPPU computing power is insufficient, we may either stall the 2-D computing array or discard some of the faulty PEs, which will degrade the performance of the DLAs. In addition, the DPPUs can also be utilized to sequentially scan and check the computing results of all the PEs in the 2-D computing array such that faulty PEs in the 2-D computing array can also be detected at runtime.
The contributions of this work are summarized as follows. The second and the third contributions are mainly extended on top of DPPU proposed in the conference paper [8] [23]. To analyze the influence of hardware faults on the deep neural network models, the authors in [15] conducted comprehensive experiments and the experiment results show that hardware faults can lead to considerable prediction accuracy drop. For the TIMIT speech recognition task, the accuracy drops from 74.13% to 39.69%. The accuracy loss is relevant to various design factors including the quantization, data format, and network architecture. It remains rather challenging to ensure resilient deep neural network execution on DLAs.
To alleviate the influence of hardware faults on neural network predictions, many prior works [24] [25] [26] took advantage of the intrinsic fault-tolerance of neural network models by retraining the models for a specific fault configuration such that hardware faults can be compensated by the retrained models. Xu et al. [16] proposed an on-accelerator retraining framework to obtain models that can tolerate the random hardware faults. Li et al. [17] and Wang et al. [27] proposed to employ the model retraining for DLAs with overclocking which may incur timing errors. To train resilient deep learning models, He et al. [26] revised the loss function to obtain models that are less sensitive to the hardware faults. Unlike the above methods, Zhang et al. [28] [15] proposed to bypass the faulty PEs in the computing array with zeros or other constant values such that the faults are more easier to tolerate via retraining. Although the retraining works for many fault configurations, there is still no guarantee that the retrained models can maintain the prediction accuracy for the target mission-critical applications because of the huge number of different fault configurations. In addition, the retraining can be rather expensive especially for large datasets and models, and is required for each specific fault configuration, which further limits the adoption of the retraining approaches. Unlike the retraining-based approaches, Hanif et al. [29] proposed a training-free mapping approach to alleviate the influence of permanent PE faults in the DLA. It leverages the saliency of the neural network parameters and opts to map the salient weights to the faulty PEs as much as possible such that the negative influence of the PE faults on the neural network models is reduced. However, it works only when the fault error rate is low and can be sensitive to the fault distribution.
To enable unmodified deep learning model execution without accuracy loss, an intuitive approach is to develop reliable DLA architectures with the conventional TMR and DMR approaches to tolerate the hardware faults, but these approaches incur substantial hardware resource consumption. In this case, the authors in [20] [19] [30] proposed to add redundant PEs to the large regular homogeneous computing array and each redundant PE can be shared by a group of PEs with distinct redundancy sharing methods such as row redundancy (RR), column redundancy (CR), and diagonal redundancy (DR). For the RR and CR, each row/column of the PEs share the same set of redundant PEs. For the DR, both the row and column of PEs corresponding to a diagonal location in the computing array share the same set of redundant PEs. Since the redundant PEs are shared by a group of homogeneous PEs, hardware resource consumption can be greatly reduced compared to DMR and TMR. Nevertheless, the faulty PEs may not be evenly distributed or even clustered across the computing array [31]. As a result, these approaches may fail to recover the computing array when the number of faulty PEs in each shared region such as a row or a column of PEs exceeds the the number of shared redundant PEs in the region. Thereby, the utilization of the redundant PEs can be affected by the fault distribution dramatically. More redundant PEs must be designed to ensure reliable execution. Otherwise, the performance can degrade dramatically when the faulty PEs are discarded due to the insufficient redundant PEs. Thereby, more efficient computing array architectures are required for the highly resilient DLA designs.
On top of the redundancy-based fault-tolerant DLA designs, there are also many other different fault-tolerant architecture designs. The authors in [32] [33] proposed a spatial and temporal checksum to protect full connection and convolution layers in deep neural network models. Zhang et al. [34] proposed a parallel stochastic computing(SC)-based neural network accelerator purely using bitstream computation by fully exploiting the superior fault tolerance of SC mainly for ternary neural networks. Li et al. [35] proposed an error detecting scheme to locate incorrect PEs of the DLA and gave an error masking method to achieve fault-tolerance. Hamid Reza Mahdiani etc. [36] proposed to relax the fault-tolerance of the VLSI implementation by employing TMR to only the computation of the most important bits such that the hardware overhead is reduced and the critical path latency is improved. Nevertheless, these approaches either require model retraining or can be sensitive to the fault distribution.

III. BACKGROUND AND MOTIVATION
A typical DLA with 2D computing array is shown in Figure 1. The computing array is composed of multiple ho- Neural network operations such as convolution can be mapped to the computing array and executed in lock-step manner. To ensure a high-throughput neural network processing, the input features, weights and output features must be stored in the on-chip buffers to avoid external memory access stalls. In this work, we adopt a widely used output stationary dataflow for the neural network execution [13]. The accumulation of each output feature stays stationary in a PE. The partial sum are stored in the same register file for accumulation to minimize the accumulation cost. In summary, each PE is responsible for the calculation of a single output feature and PEs in the same column calculate different output features in the same output channel. With the compact dataflow, the 2-D computing array can be fully utilized given limited on-chip buffer bandwidth provision when the neural network models are deployed on it.
A. Deep Learning Accelerator Architecture

B. Influence of Faulty Computing Array
To analyze the influence of hard errors on the above 2-D computing array, we inject random stuck-at bit errors to the registers of the PEs in a 32 × 32 computing array. We use bit error rate (BER) as the fault injection rate metric [11] [37] [38], which refers to the total number of bit errors over the total bit number of the registers in the computing array. To facilitate the error characteristic of the 2-D computing array, we convert the BER to PE error rate (PER) instead. Both the input features and weights are 8-bit fixed point, so the input registers and the weight registers are set to be the same data width accordingly. The intermediate register in the PEs is set to be 16-bit and the accumulator in the PEs is set to be 32-bit in case of computing overflow. Thereby, there are 64 bit registers in total in each PE. While any persistent bit error in a PE is considered as an PE error, PER can be calculated using BER with (Equation 1). Basically, it means that the PE is correct only when none of the bit registers are wrong. Otherwise, the PE will be faulty.  We had random stuck-at bit errors injected to a DLA simulator implemented according to the architecture described in Figure 1 for the fault analysis. We took Resnet18 pre-trained on ImageNet [39] as an example and had it implemented on the accelerator with random faults. In this case, we generated 50 random fault configurations and evaluated the prediction accuracy under different PER setups. The experiment result is shown in Figure 2. It reveals that the prediction accuracy varies dramatically across the different fault configurations. When the PER is higher than 1%, the prediction accuracy of the model mostly degrades to zero. Moreover, we notice that the prediction accuracy may also drop considerably in some of the fault configurations even under very low PER. It indicates that the model accuracy depends on not only the PER but also the fault distribution. Thereby, protecting the computing array is required for mission-critical applications despite the fault injection rate.
In addition, we further evaluated the classical hardware redundancy strategies, i.e. RR, CR, and DR for the regular 2-D computing array and measured the fully functional probability of the computing array under different PER setups. The evaluation result is shown in Figure 3. It can be observed that these classical redundant design approaches can hardly mitigate all the faulty PEs even when the PER is around 1% which indicates that there are only 10 faulty PEs on average. In contrast, the number of redundant PEs is 32, which is much larger than the number of faulty PEs. It demonstrates that the redundant PEs cannot be fully utilized by these redundancy strategies because of the unevenly faulty PE distribution. The situation further deteriorates with the increase of the PER, which can be rather risky for the mission-critical AI applications.

IV. HYCA FOR FAULT-TOLERANT DLAS
In this section, we will present an overview of the proposed hybrid computing architecture (HyCA) for fault-tolerant DLAs first. Then, we will illustrate the dataflow for the fault mitigation, HyCA microarchitecture, and fault detection with HyCA respectively.

A. HyCA Overall Architecture
In order to tolerate various fault configurations with a unified computing architecture, we propose a hybrid computing architecture (HyCA), which has a dot-production processing unit (DPPU) seated along with the regular 2-D computing computing array, to recompute all the operations mapped to the faulty PEs in arbitrary locations of the computing array as shown in Figure 4. While the 2-D computing array has each PE to calculate the different output features sequentially given the output stationary dataflow [13] and the DPPU has all the PEs to compute a single output features in parallel, they have distinct read patterns of the input features and weights from the corresponding on-chip buffers. More specifically, the 2-D computing array needs to read an array of input features in the same row and channel in each cycle while DPPU needs to read an array of input features aligned in channel dimension in each cycle. Thereby, the on-chip buffers cannot fulfill the read operations of the two computing units at the same time due to the limited read ports and distinct data layout requirements. To make sure that the normal 2-D array processing will not be affected by the DPPU recomputing, the on-chip buffer design remains unchanged. In this case, DPPU cannot read the required weights and input features aligned in channel dimension if it starts the recomputing at the same time with the 2-D computing array.
To address the problem, we have the input features and weights buffered in an input register file (IRF) and a weight register file (WRF) respectively while they are read for the  2-D computing array processing. Meanwhile, we have the recomputing delayed until there are sufficient inputs and weights ready for the recomputing. Accordingly, the delay must be larger than or equal to the number of weights required by DPPU data consumption in a single cycle to ensure DPPU can be fully utilized. As the DPPU may recompute operations on any PE in the 2-D computing array, the delay also needs to be larger than or equal to Col when the last column of the PEs obtain the weights passed from the first column of PEs. Suppose D represents the delay, then D ≥ Col. Note that Col refers to the column size of the 2-D computing array. In this work, we organize IRF and WRF in Ping-Pong manner to ensure that the 2-D computing array can continue the normal dataflow without any stall during the DPPU recomputing. As the DPPU conducts the output feature calculation in parallel, DPPU can always finish the recomputing of the operations mapped to the faulty PEs before the Ping-Pong register files swap with each other when the DPPU size does not exceed the number of the faulty PEs. Note that DPPU size refers to the number of multipliers in DPPU. Since the peak computing power of DPPU equals to that of the 2-D computing array when configured with the same number of PEs, DPPU size is comparable to the 2-D computing array size and can also be used to represent its computing power. This also explains why DPPU can always finish the recomputing tasks before new weights and inputs are ready when DPPU size is larger than the number of faulty PEs in the 2-D computing array.
In addition, we have a fault PE table (FPT) to record the coordinates of the faulty PEs in the 2-D computing array which can be usually obtained with a power-on self-test procedure. With the coordinates, an address generation unit (AGU) is used to generate the read addresses and instruct the DPPU to read the right input features and weights from the register files. Moreover, AGU also determines the addresses to the output buffer for the overlapped writes of the recomputed output features. Similar to the IRF and WRF, there is also a Ping-Pong register file called ORF for the DPPU outputs and it is utilized to pipeline the DPPU recomputing and the write from DPPU to the output buffer.
While DPPU can be utilized to calculate any output features mapped to the 2-D computing array, we can also use DPPU to check whether the calculation of an output feature in the 2-D computing array is correct, which can be used to detect the wear-out or aging induced persistent errors at runtime. If the computing results obtained from the 2-D computing array and DPPU do not match, it indicates that the corresponding PE in the 2-D computing array is faulty as DPPU with much smaller sizes compared to the 2-D computing array can be easily protected with much less overhead and is usually considered to be correct. By changing the fault PE table and scanning the computing of all the PEs in the 2-D computing array sequentially, we can detect the PE faults at runtime without affecting the 2-D computing array processing. Basically, the redundant recomputing mechanism can be mostly reused by the fault detection. And we only need a tiny fault detection module to conduct the scanning of the 2-D computing array and the comparison to the DPPU processing. Details of the fault detection module will be illustrated in the rest of this section.

B. HyCA Dataflow for Fault Mitigation
To further illustrate the dataflow in HyCA especially the redundant computing unit DPPU, we take HyCA with a 32 × 32 2-D computing array and three faulty PEs as an example. DPPU in the HyCA has 32 PEs included. The example is shown in Figure 5. Suppose c and k represent the input channel depth and the convolution kernel size respectively. It takes a PE in the 2-D computing array k × k × c cycles to produce a convolution output. Without loss of generality, assume that the example starts at Cycle k × k × c − 1 when the first column of PEs complete a set of output feature calculation. The 2-D computing array occupies the output buffer until the last column of PEs complete the output feature data. Afterwards, the 2-D computing array may start to compute the new output features, but it usually takes k × k × k × c cycles to complete with the output stationary dataflow. In this case, DPPU starts to use the output buffer and update the recomputed results to the output buffer without write conflicts. The processing steps are detailed as follows.
1) At cycle k × k × c − 1, the first column of PEs produce a column of output features accordingly and pass the weights to the second column of PEs. At the same time, the weights and input features used in the first column of PEs are stored in the corresponding Pong register file. 2) At cycle k × k × c, the first column of PEs write the calculated output features to the output buffer and start the calculation of new output features. Since PE(1,0) is faulty, the computing result written to the output buffer from this PE as marked with red color was wrong. While the input features and weights that are used for the output feature calculation on PE(1, 0) remain stored in the IRF and the WRF respectively, they will be read to the DPPU for the recomputing at this cycle. 3) At cycle k × k × c + 31, the Pong WRF and the Pong IRF are filled with the newly incoming weights and inputs, and they will be kept for 32 cycles. Weights and input features coming in next cycle will overwrite the data in the Ping WRF and the Ping IRF respectively. Thereby, DPPU must finish the recomputing that depends on the weights stored in the Ping WRF and IRF at this cycle. Otherwise, the data in the Ping register files will be overwritten. Afterwards, the processing repeats from the first processing step for another 32 cycles until the end of the convolution calculation. 4) At cycle k × k × c + 32, DPPU has the recomputed output features in the output register file (ORF) written to the output buffer with a byte mask such that only the recomputed output feature is updated. Meanwhile, it starts to recompute the latest set of output features that are mapped to the faulty PEs in the 2-D computing array. As each output feature calculation is mapped to a single PE in the 2-D computing array using the classical output stationary dataflow, it takes c × k × k cycles to complete an output feature calculation. As c×k×k is usually larger than 32, and the output buffer will be occupied for only 32 cycles during each set of output feature calculation, the recomputed output feature data can be updated to the output feature buffer without conflicts. 5) At cycle k×k×c+34, because there are only three faulty PEs in the 2-D computing array, it takes the DPPU three cycles to have the recomputed results overwritten from the ORF to the output buffer. 6) From cycle k × k × c + 35 to k × k × c + c, both the 2-D computing array and the DPPU conduct the partial convolution locally, so the output buffer port is idle before the first column of PEs complete the new output feature calculation. As shown in Figure 5, the overall processing is conducted iteratively. Each iteration includes a set of complete output feature calculation and it can be divided into three phases, i.e. 2-D array write, DPPU write, and idle from the perspective of the output buffer status. While each PE produces an output feature data per iteration and a PE conducts one MAC per cycle, the processing time of an iteration is T iteration = c × k × k. For the 2-D computing array write, it takes T 2D arrary write = D cycles per iteration where D refers to the number of cycles that DPPU delays after the 2-D computing array processing. For the DPPU write, it needs T DP P U write = f ault P E num cycles as the DPPU updates the recomputed output features sequentially. However, DPPU recomputing does not have to start after the entire computing of the output features on the 2-D computing array. Instead, it is pipelined with the 2-D computing array but only D cycles slower. Thus, the weights and the input features consumed by the 2-D computing array must be fully accommodated by the Ping-Pong register files during the D cycles. Accordingly, the depth of weight and input feature Ping-Pong register files is set to be 2×D ×Row.
To minimize the register file overhead, we set D = Col.
As the average throughput of a PE in the 2-D computing array is the same with that in the DPPU, each multiplier in the DPPU can be used to repair a faulty PE in the 2-D computing array. Thereby, the DPPU size essentially represents the capability of the fault tolerance of the proposed HyCA without performance penalty. When the number of faulty PEs is larger than the DPPU size, we seek to preserve the computing power as much as possible without altering the target neural network models. To that end, we discard the faulty PEs that cannot be repaired due to the lack of the computing redundancy in the DPPU. As it is usually inefficient to compile and deploy the neural network models to a computing array with irregular row sizes which can cause both the irregular on-chip buffer accesses and external memory accesses, we choose to discard the columns with unrepaired faulty PEs and the columns that are disconnected from the input/weight/output buffers. Moreover, HyCA can repair any faulty PEs in the 2-D computing array, so it offers more flexibility to prioritize the faulty PEs for repairing such that the surviving computing array can be maximized especially when there are insufficient redundant PEs. In this work, the maximum remaining computing array can be obtained simply by assigning higher repairing priority to the faulty PEs on the left, which ensures that the surviving computing array is connected to the on-chip buffers.

C. HyCA Microarchitecture
In this section, we will illustrate the major components of HyCA added to the baseline DLA and they include the DPPU, register files, and the fault PE table (FPT). FPT keeps the coordinates of the faulty PEs that will be repaired by the DPPU. As the maximum number of faulty PEs that can be tolerated without performance penalty is determined by the DPPU size, FPT is configured with DP U size entries accordingly. Address generation unit (AGU) is a piece of control logic that generates the access addresses of the weight register file, input register file, and output register file based on the FPT for the recomputing of the DPPU. The structures of FPT and AGU are simple and we will not dwell on it. In contrast, the DPPU and the register files are relatively more complex, and they dominate the hardware overhead. Thus, they will be detailed in the rest of this section.
1) Dot-Production Processing Unit (DPPU): DPPU is utilized for the dot-production and it consists of a set of multipliers as well as an adder tree that is used to aggregate all the multiplication results in a pipelined manner. It is mainly used to recompute the output features that are mapped to the faulty PEs in the 2-D computing array. An intuitive implementation is to construct a single unified dot-production unit which has both the input features and weights read from the corresponding register files in a single cycle and processed in parallel. As DPPU starts Col cycles later after the 2-D computing array, each faulty PE in the 2-D computing array has Col weights and input features multiplied and accumulated. Accordingly, Col weights and input features can be extracted for the recomputing on DPPU for each faulty PE in the 2-D computing array. In order to make best use of the PEs in the DPPU, the Col weights and input features must be fully distributed to the DPPU. If the entire DPPU is organized into a unified dot-production unit, the size of the DPPU is rather limited, which hinders the scalability of the DPPU. For instance, when Col is set to be 32 and DPPU size is set to be 24 or 48, the computing mapped to a single faulty PE cannot be perfectly mapped to the DPPU, which will lead to the under-utilization of the DPPU. To address this problem, we propose to divide the PEs in the DPPU into multiple smaller groups and each group can conduct the dotproduction independently. As the number of PEs in each group gets smaller, they are more likely to be fully utilized by the computing mapped to a faulty PE. As shown in Figure 6, each group includes 8 PEs and it completes the computing of a faulty PE in 4 cycles when Col is set to be 32. In this case, the DPPU size can be scaled conveniently. At the same While the DPPU is used to recompute all the faulty operations in the 2-D computing array, it must be resilient enough to ensure the functionality. Otherwise, a single fault in the DPPU may corrupt the whole accelerator. To improve the resilience of the DPPU, we add redundant PEs to the DPPU as shown in Figure 6. Basically, the multipliers used in the DPPU are divided into groups and each group is equipped with a redundant multiplier. Instead of having the redundant multiplier shared by all the multipliers in the group, we have the redundant multiplier and the multipliers in the group connected in a directed ring topology and each multiplier can be configured to replace its downstream neighboring multiplier. When any of the multiplier fails, it can be replaced by its upstream multiplier immediately. Compared to the shared redundancy design, this approach can avoid high fan-out connections to the redundant multipliers. Similarly, we also have the adders in the adder tree protected with the same redundancy design approach.
2) Register Files: The IRF and the WRF are used to back up the data read from input buffer and weight buffer, and then supply the data to the DPPU for the recomputing. As the 2-D computing array and the DPPU have different dataflows for the neural network computing, the weight register file is written in column-wise manner but read in row-wise manner as shown in Figure 7. When the DPPU is split into multiple groups, these groups will be responsible for different faulty convolution operations and they need to read different rows of WRF and IRF at the same time. Although a straightforward multi-port register file can fulfill the concurrent register file read, it will induce substantial hardware overhead according to [40], [41].
While we observe that each computing group in DPPU has only a small number of PEs and they cannot consume a single row of inputs and weights in a single cycle. As a result, the straightforward multi-port register file actually has the bandwidth wasted. With this observation, we also have the register files split into groups in row direction such that each group of the register file can be read independently by the corresponding computing group in the DPPU. In this case, each register file has only a single read port, but each computing group can only read a segment of the data in the register files as indicated by the blue color. While an output feature recomputing on DPPU needs an entire row of data in the register files, we have each row of the registers organized as a circular shift register. With the shift register, different segments of the data in a row can be obtained by the corresponding computing group in DPPU in a few cycles. At the same time, the amount of data fed to each computing group can be fully utilized. Moreover, we notice the read port data width of the register files is not necessarily equal to the DPPU size. When the DPPU size is larger than the register file data width, more read ports can be added to some of the register file groups rather than the entire register file. When the DPPU size is smaller than the register file data width, some of the register file groups do not even need a read port as shown in Figure 7. Thereby, the DPPU size can be scaled conveniently and it will not be limited by the register file sizes.

D. Fault Detection with HyCA
On top of the fault mitigation, DPPU can also be utilized to conduct fault detection at runtime. The basic idea is to have the DPPU to recompute the operations on a PE in the 2-D computing array. Then, we have the computing results compared to check if the PE in the 2-D computing array is faulty. By scanning all the PEs in the 2-D computing array sequentially, we can determine if the 2-D computing array is faulty. While the DPPU always starts the recomputing Col cycles later, the computing result of a PE to be checked is already updated or written to the output buffer when the DPPU completes the recomputing. To address the problem, we have the computing results to be checked buffered in a checking list buffer (CLB) as shown in Figure 8. As the fault detection scanning is conducted sequentially, a simple on-chip buffer can fulfill the requirements. When the DPPU completes the recomputing, the fault detection module can have the results compared with that stored in the CLB.
As hard faults in a PE can usually lead to computing errors of most of the computation, we do not have to compare the final output feature computing results for the fault detection.
Instead, we can compare the partial computing results of a PE for the fault detection such that the fault detection can be faster and more efficient. Since the DPPU conducts the processing in parallel, we have the partial computing result (PR) produced by a single DPPU group in a single cycle compared in this work. Different from the DPPU, PEs in the 2-D computing array have the computing results accumulated continuously before the entire output feature processing is completed. Suppose only one DPPU group is reserved for the runtime fault detection and the DPPU group includes S PEs. To enable the comparison for the fault detection, we have both the base accumulated results (BAR) and the accumulated results (AR) calculated S cycles later stored in the CLB. In the next cycle, another pair of the BAR and the AR from different PEs will be stored accordingly. As the weights and input features are stored in their register files for only Col cycles, we only have Col ARs and BARs stored in the CLB. While the CLB is also organized in Ping-Pong manner, the total size of the CLB is 4 × W × Col Bytes where W denotes the width of the accumulator in PEs.
According to the recomputing dataflow, the reserved DPPU group performs the recomputing Col cycles later. Unlike the fault-tolerance oriented recomputing, the fault-detection oriented recomputing only conducts the dot-production of S weights and input features rather than Col weights and input features. The results of the DPPU will be compared with the corresponding results stored in the CLB for the fault detection. Basically, AR will be compared to the addition of PR and BAR. When a faulty PE is detected, the faulty information i.e. the fault PE row index and column index will be updated to the FPT. One comparison can be done per cycle, so it takes the fault detection module Col cycles to complete the comparison with the stored results in CLB. Accordingly, it takes the fault detection module Row × Col + Col cycles to complete the fault detection of the whole 2-D computing array. When a DPPU group has more PEs included, it needs to check a partial result with more computation. As a result, the fault detection time is independent with the number of PEs in the DPPU group. While the fault detection time is already much smaller than the processing time of a normal neural network layer, it is fast enough to detect the computing errors. To avoid the frequent fault detection, the fault detection module can be activated periodically in a larger time range depending on the requirements of the applications. In addition, the fault detection module can reuse the majority of the fault recovery design, it induces only some simple controlling logic and a small CLB, and consumes negligible chip area.

V. EXPERIMENT
A. Experiment Setup 1) Accelerator Configurations: The proposed deep learning accelerator with HyCA is implemented in Verilog and synthesized with Design Compiler under TSMC 40nm technology. The computing array size and DPPU size are set to be 32 × 32 and 32 respectively. The input feature buffer size is 128KB, output feature buffer size is 128KB and the weight buffer size is 512KB. The computing of DPPU is delayed by D = 32 cycles after the 2-D computing array, so both the weight register file size and the input register file size are set to be 2 × 32 × D = 2048 i.e. 2KB. The output register file in DPPU is 64-byte. The fault PE table size is 32 × 10bits. Each entry of the table includes 5-bit row index and 5-bit column index of a faulty PE. Both the data width of weights and input feature data is 8-bit. To ensure the resilience of the DPPU, we have every four multipliers in the DPPU grouped and equipped with a redundant multiplier, and every three adders in the DPPU grouped and protected with a redundant adder. For the 2-D computing array, we have three classical redundancy approaches including RR, CR, and DR implemented and each redundancy implementation is equipped with 32 redundant PEs.
2) Fault Models: To evaluate the reliability of the redundancy designs comprehensively, we have two different fault distribution models including the random distribution model and the clustered distribution model implemented. For the random distribution model, the faults are randomly distributed across the entire computing array. For the clustered distribution model which is usually used to characterize the manufacture defects, the faults are more likely to be close to each other and the model proposed in [42] is applied in this work. Meanwhile, we notice that the influence of hardware faults is related with the fault distribution, so we generate 10000 configurations randomly for each fault injection rate and average the evaluation in the experiments.
As hard errors are mainly caused by the manufacturing defects, aging, and wear-out, which can be affected by many complex factors such as application requirements, working status, and the fabrication, there is still a lack of references investigating the practical PER setup and prior fault-tolerant designs typically have distinct error rate setups [28], [29], [43], [44]. In this case, we evaluate the hard error rate in a large range and seek to demonstrate when we can ensure reliable computing. Then, we expect the users to choose the target hard error rate for their specific fault-tolerant designs. In addition, since we mainly focus on the reliability of the regular 2-D computing array in a deep learning accelerator, we use PE error rate (PER) as the fault injection metric similar to the works in [45] and [46]. Meanwhile, we notice that bit error rate (BER) that refers to the number of bit errors over the total number of memory bits is directly related with definition of the hard error on chips, and it has been widely utilized as a metric for fault analysis in many prior works [11] [37] [38]. Thus, we convert the BER to PER with (Equation 1) mentioned in Section III assuming that any bit error in a PE will cause the PE failure. While BER typically ranges from 1 × 10 −7 to 1 × 10 −3 , PER ranges from 0% to 6% according to the conversion.
3) Neural Network Benchmark: To evaluate the performance of a typical DLA with the proposed fault-tolerant HyCA, we have a set of representative neural network models including Alexnet, VGG, Resnet, and YOLO used as the benchmark. Alexnet, VGG and Resnet are classical models used for image classification, while YOLO is mostly used for object detection. All the models are pre-trained on ImageNet. We measured the performance of the benchmark on the DLAs with different redundancy design approaches using Scale-sim [47]. Since Scale-sim is relatively slow, it is difficult to obtain the performance of all the random fault configurations directly.
In this experiment, we determined the final valid computing array setups of all the fault configurations and performed the simulation on only the unique computing array setups. As many fault configurations lead to the same computing array setups eventually, this approach greatly reduces the simulation time. Finally, we averaged the resulting performance based on the generated fault configurations. Figure 9 illustrates the chip area of the DLAs with different redundancy approaches including RR, CR, DR, and HyCA. Particularly, we have three HyCA-based designs with different DPPU sizes compared. The DPPU size of HyCA24, HyCA32, and HyCA40 is 24, 32 and 40 respectively. According to the comparison Figure 9, the HyCA-based designs exhibit much less redundancy overhead compared to the classical redundancy designs. The redundancy overhead of the HyCAbased designs mainly consist of the redundant PEs and the register files, while the redundancy overhead of the RR-based, CR-based, and DR-based designs are mainly attributed to the MUX and the redundancy PEs. As the 2-D computing array size is 32 × 32, the number of redundant PEs in RR-based, CR-based, and DR-based designs is the same and the chip area caused by the redundant PEs is also equal. While HyCA has different PE structures, i.e. independent multipliers and adders rather than MACs, and additional redundant PEs, the chip area of the redundant PEs is larger given the same DPPU sizes. In contrast to the chip area of the redundant PEs in HyCA, the added small Ping-Pong register files in HyCA consumes much less chip area. Different from HyCA, RR, CR and DR include a large number of MUX to enable the replacement of faulty PEs with the redundant PEs. These MUX take up substantial chip area and dominate the redundancy overhead.

C. Reliability Comparison
To evaluate the reliability of the DLAs, we propose two metrics that can be applied for different applications. One of them is the fully functional probability and it shows the probability that the DLA can be fully functional without any performance penalty. It is preferred by the mission-critical applications that do not allow any performance degradation nor model modification because any system modification may require expensive and lengthy safety evaluation and certification. The experiment is shown in Figure 10. It shows that HyCA outperforms the three classical redundancy approaches and the advantage gets enlarged under the clustered fault distribution.
The main reason is that each redundant PE in RR, CR and DR can only be utilized to replace a single faulty PE in a row, a column, and a row-column pair respectively. When multiple faults occur in the same protected region, these redundancy approaches fail to recover the faulty 2-D computing array and the design will not be fully functional. Unlike these classical redundancy approaches, HyCA allows arbitrary faulty distribution and can perfectly repair the computing array as long as the number of faulty PEs in the 2-D computing array does not exceed the DPPU size. Thereby, the fully function probability of HyCA is not sensitive to the fault distribution models. As DPPU size is set to be 32 and the 2-D computing array size is 32 × 32 in this example, the fully functional probability drops to 0 immediately when the number of fault PEs exceeds 32 at 3.13% PER. As the PEs in the DPPU can also be faulty, the fully functional probability of HyCA starts to drop when the number of faulty PE is close to 32 and the PER is slightly lower than 3.13%. The other metric is the normalized remaining computing power and it refers to the percentage of the remaining computing array size over the original 2-D computing array size. This metric is particularly important for the non-critical applications that do not require fully functional accelerators and allow the accelerators to be degraded, because the remaining computing array size determines the theoretical computing power and affects the performance of the deployed neural network models directly. In this work, we apply the acceleration degradation strategy mentioned in the end of Section IV-B and discard the faulty PEs in the granularity of a column when the redundant PEs are insufficient to mitigate all the faulty PEs. Although more aggressive degradation approaches are possible to achieve larger computing power, this approach is applied for more efficient model compilation, hardware implementation and memory accesses. Figure 11 reveals the computing power comparison of the different redundancy approaches. It can be observed that HyCA shows significantly higher computing power under all the different PER setups and the advantage also enlarges with the increase of the PER. This is mainly brought by the fault recovery flexibility of the HyCA that allows the DPPU to select the most critical faulty PEs to repair when the redundant faulty PEs are insufficient. Note that the most critical faulty PEs refer to the ones that can maximize the remaining 2-  Fig. 12. Normalized performance of DLAs with different redundancy approaches D computing array. By optimizing the faulty PE mitigation order, the remaining computing array can be larger. In contrast, each redundant PE can only repair a limited subset of the faulty PEs for the RR, CR and DR. There is little space left to optimize the faulty PE mitigation order. Thereby, the remaining computing power of RR, CR, and DR is much lower. As we choose to discard the faulty PEs that are failed to be repaired in the granularity of a column, RR cannot effectively shift the faulty PEs to a different column and has to discard the column whenever there are more than one faulty PEs. As a result, RR shows the lowest computing power even when the number of redundant PEs is the same with the other redundancy approaches.

D. Performance Comparison
In order to evaluate the performance of the DLAs protected with the different redundancy approaches, we have the neural network benchmark deployed on the DLAs with Scale-Sim. The performance is normalized to that of the DLA protected with RR and the experiment result is shown in Figure 12  be found that HyCA achieves much higher performance than the other redundancy approaches especially under relatively higher PER, which is roughly consistent with the experiment in Figure 11 though the neural networks also affect the performance. Particularly, the performance speedup goes up to 9X when the PER is around 6% under the random fault distribution. The underlying reason for the higher performance speedup at higher PER is that higher PER indicates more faulty PEs in the 2-D computing array and leaves larger optimization space for the HyCA. Another observation is that the performance gap between HyCA and the other redundancy approaches is much smaller than the computing power gap. For instance, the computing power of HyCA is around 25X higher than RR when PER is 6% under the random fault injection while the performance speedup is only 9X. This is mainly attributed to the fact that the neural network runtime varies dramatically under different computing array sizes as shown in Figure 13. When the remaining computing array size is large at lower PER, the runtime decreases much slower with the increase of the computing array size. In addition, some of the neural networks like VGG include some full connection layers that fail to make best use of the computing array. In fact, only a single column of PEs is used for the full connection operations given the output stationary dataflow and the larger remaining computing array in HyCA is underutilized, which also undermines the performance speedup.

E. Redundancy Design Scalability Analysis
As different applications may have distinct requirements of reliability and may also work under different fault environments, a scalable redundancy design can greatly alleviate the reliability design problems. In this subsection, we will investigate and compare the scalability of the different redundancy approaches under different computing array sizes. As the fully functional probability and the computing power is roughly positively related, we only use the fully function probability as the metric for the scalability evaluation to save the space. The number of redundant PEs in RR and CR is consistent with the corresponding computing array row size and column size respectively. Although the number of redundant PEs in DR is also equal to the diagonal size of the computing array, it cannot be directly applied to a nonsquare computing array. In this experiment, we divide the non-square computing array into multiple square computing arrays and apply the diagonal redundancy approach to each sub computing array independently. The number of redundant PEs in HyCA is set to be Col to ensure a fair comparison where Col refers to the column size of the computing arrays.
The experiment result is shown in Figure 14. It can be observed that the fully functional probability of RR, CR and DR under different PER setups vary dramatically when the computing array size changes. CR and DR have the same amount of redundant PEs on the different 2-D computing array sizes. Basically, the redundancy intensity, i.e. the average redundancy per PE in the 2-D computing array, vary dramatically across the different computing arrays. Thus, the fully functional probability curves are different accordingly. The number of redundant PEs for RR scales with the three specific computing arrays, but the fully functional probability curves are closer to each other but remains different due to the fault distribution variations. In general, the classical redundancy approaches do not scale well and the sensitivity to the fault distribution further aggravates the scalability problem. In contrast, the proposed HyCA exhibits much better scalability and shows consistent fault-tolerance capability under different computing array sizes and fault distribution models.
We also evaluated the scalability of the two different DPPU implementations, i.e. the Unified DPPU and the Grouped DPPU under different DPPU sizes. We scale the DPPU sizes from 16 to 48 and fix the computing array size to be 32 × 32. The experiment result is shown in Figure 15. It can be observed that the Grouped DPPU scales strictly with the DPPU sizes. The Unified DPPU scales when the DPPU size is set to be 16 and 32, but it does not scale when the DPPU size is set to be 24, 40, and 48. The main reason is that the Unified DPPU needs to read from input and weight register files in which the data are aligned with the column size of the computing array. More specifically, the input features and weights are aligned to 32, i.e. the column size. When the DPPU size is larger than 32 and it cannot be perfectly split by 32, the Unified DPPU cannot be fully utilized due to the lack of the sufficient input data. When the DPPU size is smaller than 32 and 32 data cannot be divided perfectly for the Unified DPPU processing, the Unified DPPU also suffers underutilization and leads to unsatisfactory scalability in this occasion. Different from the Unified DPPU, the Grouped DPPU can be utilized with smaller granularity and is able to make full use of the aligned data from both the input and weight register files.
As the hardware overhead of DPPU is mainly caused by the redundant PEs according to the experiment in Section V-B, the hardware overhead of both DPPU implementations scales with the DPPU size accordingly.

F. Fault Detection Analysis
With the proposed fault detection approach, PE faults can be detected at runtime. Since the proposed fault detection module essentially scans all the PEs in the 2-D computing array sequentially, we mainly evaluate the fault detection scanning time under different computing array sizes, and compare the fault detection time to the corresponding neural network processing time. Basically, we want to determine if a runtime persistent fault can be detected before a neural network layer is computed. And we take the percentage of the neural network layers that can be detected during the layer processing as a metric to evaluate the fault detection capability. The experiment result is shown in Table I. It can be observed that all the faults in the 2-D computing array can be detected during the execution of each neural network layer when the 2-D computing array size is smaller than or equal to 64 × 64. When the 2-D computing array size reaches 128 × 128, the processing time of some small neural network layers can finish the processing before the fault detection module scan the entire 2-D computing array. In this case, we may have to add more DPPU groups for the fault detection.
The fault detection module mainly includes a check list buffer (CLB) and some control logic. The CLB is Col×W ×4 bytes and dominates the chip area, but it is only Row/(2×W ) (i.e. 1/4 when Row = 32 and W = 4) of the input register file. Thus, the CLB overhead is much smaller than the input register file let alone the redundant PEs. Thereby, the chip area of the fault detection module is negligible.

VI. CONCLUSION
The reliability of DLAs is of vital importance to the mission-critical AI applications. Prior redundancy design approaches for the regular computing array such as RR and CR greatly reduce the hardware overhead compared to the classical TMR approaches, but they are rather sensitive to the fault distribution and fail to work especially when the faults are unevenly distributed. To address this problem, we propose a hybrid computing architecture (HyCA) and have a DPPU to recompute all the operations mapped to the faulty PEs in the 2-D computing array. When the number of faulty PEs in the 2-D computing array is less than the DPPU size, HyCA can fully recover the 2-D computing array despite the fault distribution. Even when the fault error rate further increases, DPPU can still be used to repair the most critical PEs first to ensure a large available computing array and minimize the performance penalty. According to our experiments, HyCA outperforms prior redundancy approaches in terms of both the fully functional probability and the computing power under different fault distribution models. In addition, HyCA can also be reused for the fault detection at runtime and the experiment result shows that the entire 2-D computing array can be scanned and detected before a neural network layer completes its execution in most cases.


Quantifying Reproducibility in NLP and ML

Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so far emerged. The assumption has been that wider scientific reproducibility terminology and definitions are not applicable to NLP/ML, with the result that many different terms and definitions have been proposed, some diametrically opposed. In this paper, we test this assumption, by taking the standard terminology and definitions from metrology and applying them directly to NLP/ML. We find that we are able to straightforwardly derive a practical framework for assessing reproducibility which has the desirable property of yielding a quantified degree of reproducibility that is comparable across different reproduction studies.

Introduction
Reproducibility of results is coming under increasing scrutiny in the machine learning (ML) and natural language processing (NLP) fields, against the background of a perceived reproducibility crisis in science more widely (Baker, 2016), and NLP/ML specifically (Pedersen, 2008;Mieskes et al., 2019). There have been several workshops and checklist initiatives on the topic, 1 conferences are promoting reproducibility via calls, chairs' blogs, 2 and special themes, and the first shared tasks are being organised, including REPROLANG'20 (Branco et al., 2020) and ReproGen'21 (Belz et al., 2020a). The biggest impact so far has been that sharing 1 Reproducibility in ML Workshop at ICML'17, ICML'18 and ICLR'19;Reproducibility Challenge at ICLR'18,ICLR'19,NeurIPS'19,and NeurIPS'20; reproducibility track and shared task (Branco et al., 2020) at LREC'20; reproducibility programme at NeurIPS'19 comprising a code submission policy, a reproducibility challenge for ML results, and the ML Reproducibility checklist (Whitaker, 2017), later also adopted by EMNLP'20 and AAAI'21.
2 https://2020.emnlp.org/blog/ 2020-05-20-reproducibility of code, data and supplementary material providing increasingly detailed information about data, systems and training regimes is now expected as standard. Yet early optimism that "[r]eproducibility would be quite easy to achieve in machine learning simply by sharing the full code used for experiments" (Sonnenburg et al., 2007) is now giving way to the realisation that even with full resource sharing and original author support the same scores can often not be obtained (see Section 2.2).
Despite a growing body of research, no consensus has so far emerged about standards, terminology and definitions. Particularly for the two most frequently used terms, reproducibility and replicability, divergent definitions abound, variously conditioned on same vs. different team, methods, artifacts, code, software, and data. E.g. for the ACM (Association for Computing Machinery, 2020), results have been reproduced if obtained in a different study by a different team using artifacts supplied in part by the original authors, and replicated if obtained in a different study by a different team using artifacts not supplied by the original authors. Drummond (2009) argues that what ML calls reproducibility is in fact replicability which is the ability to re-run an experiment in exactly the same way, whereas true reproducibility is the ability to obtain the same result by different means. For Rougier et al. (2017), "[r]eproducing the result of a computation means running the same software on the same input data and obtaining the same results. [...]. Replicating a published result means writing and then running new software based on the description of a computational model or method provided in the original publication". Wieling et al. (2018) tie reproducibility to "the same data and methods," and Whitaker (2017), followed by Schloss (2018), tie definitions of reproducibility, replicability, robustness and generalisability to different combinations of same vs. different data and code.
Underlying this diversity of definitions is the as- sumption that general reproducibility terminology and definitions somehow don't apply to computer science for which different terminology and definitions are needed. To the extent that terminology and definitions have been proposed for NLP/ML, these have tended to be sketchy and incomplete, e.g. the definitions above entirely skirt the issue of how to tell if two studies are or are not 'the same' in terms of code, method, team, etc.
In this paper we take the general definitions (Section 3) of the International Vocabulary of Metrology (VIM) (JCGM, 2012), and explore how these can be mapped to the NLP/ML context and extended into a practical framework for reproducibility assessment capable of yielding quantified assessments of degree of reproducibility (Section 4). We start with two informal examples (Section 2), one involving weight measurements of museum artifacts, the other weighted F1 measurements of a text classifier, and conclude with a discussion of limitations and future directions (Sections 6, 7).

Mass of a torc
The first example involves physical measurement where general scientific reproducibility definitions apply entirely uncontroversially. The torc (or neck ring) shown in Figure 1 is from the British Museum collection. 3 The information provided about it in the collection catalogue (some of it also shown in Figure 1) includes a measurement of its weight which is given as 87.20g. Museum records contain two other weight mea-3 https://www.britishmuseum.org/ collection/object/H_1991-0501-129 surements of torc 1991,0501.129, and museum staff performed four additional weighings for this study, yielding a set of seven measured quantity values for the mass of this torc, shown in the last column of Table 1. Details such as the scales used, whether they were calibrated, placed on a flat surface, etc., are not normally recorded in a museuym context, but the general expectation is nevertheless that the same readings are obtained. The values in Table 1 range from 87.2g to 92g, a sizable difference that implies there must have been some differences in the conditions under which the measurements were performed that resulted in the different measured values.
Museum records show 4 that the measurements were taken by different people (Team) at different times, up to 30 years apart (Date); staff normally place the object on ordinary electronic scales noting down the displayed number and rounding to one tenth of a gramme (Measurement method); calibration is sometimes checked with a 10g standard weight, but scales are not recalibrated (Measurement procedure). The museum catalogue also records details of a conservation treatment in 1991 for torc 1991,0501.129 (for full details see the link in Footnote 3) which included dirt removal with a scalpel, and immersion in acid (Object conditions). Table 1 provides an overview of the conditions of measurement mentioned above (Team, Date, etc.), alongside the corresponding values for condition in each of the seven measurement where available. Despite some missing information, the information indicates that the treatment for dirt and corrosion removal in 1991 led to the biggest reduction in weight, with the exact scales and use of a standard weight potentially explaining some of the remaining differences.

Weighted F1-score of a text classifier
The second example involves non-physical measurements in the form of a set of eight weighted F1 (wF1) scores for the same NLP system variant of which seven were obtained in four reproduction studies of Vajjala and Rama (2018). In the original paper, Vajjala and Rama report variants of a text classifier that assigns grades to essays written by second-language learners of German, Italian and Czech. One of the multilingual system variants, re- ferred to below as multPOS − , uses part-of-speech (POS) n-grams, and doesn't use language identity information. The information provided about it in the paper includes a wF1 score of 0.726 (Table 3 in the paper).
We have seven other wF1 scores for multPOS − from subsequent research which intended, in at least the first experiment in each case, to repeat the original experiment exactly. Arhiliuc et al. (2020) report a score of 0.680 wF1 ( Table 3 in the paper). Huber and Çöltekin (2020) report 0.681 (Table 3 in the paper). Bestgen (2020) reports three wF1 scores: 0.680 and 0.722 (Table 2 in the paper) and 0.728 (Table 5). Caines and Buttery (2020) report 2 scores: 0.680 and 0.732 (Table 4 in the paper). Four of the measured quantity values (those shown first for each team in Table 2) were obtained in conditions as close to identical as the teams could manage, but nevertheless range from 0.680 to 0.726, a difference in wF1 score that would be considered an impressive improvement in many NLP contexts.
From the papers we know that the scores were produced by five different teams (Team), up to two years apart (Date). wF1 measurements are normally performed by computing the wF1 score over paired system and target outputs, but no information about implementation of the wF1 algorithm (Measurement method) is given. In most cases, the original system code by Vajjala & Rama (V&R) was used, but Caines & Buttery (C&B) reimplemented it, and Bestgen (B) and Huber & Çöltekin (H&C) produced cross-validated results with 10 random seeds (Object conditions), rather than a single fixed seed. The measured values were all obtained in different compile-time (CT) (Object conditions), and run-time (RT) environments (Measurement procedure), except for two results where a Docker container was used. The test set was the same as the original in all but three cases (marked as B, C&B and H&C in the Inputs column) (Mea-surement procedure). Table 2 provides an overview of the conditions mentioned above (Team, Date, etc.) under which the eight measurements were obtained. Conditions can be seen as attribute-value pairs where the attribute name is here shown in the (lower) column heading, and the values for it in the table cells. We know the reason for some of the differences, because the authors controlled for them: e.g. the differences in compile time and run time environments explain the difference between Bestgen's first and second results, and reimplementation of the system code in R explains the difference between Caines & Buttery's first and second results. Despite missing information, we can see that different CT/RT, and performing 10-fold cross-validation vs. a single run with fixed seed, account for some of the remaining differences.

Comparison
In presenting and discussing the physical and nonphysical measurements and conditions in the preceding two sections, we introduced common terminology for which full VIM definitions will be provided in the next section. The intention in using the same terms and layout in Tables 1 and 2 was to bring out the similarities between the two sets of measurements, and to show that measurements can generally, whether physical or not, be characterised in the same way on the basis of the same general scientific terms and definitions.
The two fields, curating of museum artifacts and NLP/ML, also have in common that (i) conditions of measurement are not traditionally recorded in any great detail, making exact repetition of a measurement virtually impossible, (ii) despite this,  there is nevertheless an expectation that repeat measurements should yield the same results, and (iii) historically, practitioners have not cared about these issues very much.

VIM Definitions of Repeatability and Reproducibility
The International Vocabulary of Metrology (VIM) (JCGM, 2012) defines repeatability and reproducibility as follows (all defined terms in boldface, see Appendix for full set of verbatim VIM definitions including subsidiary defined terms): 2.21 measurement repeatability (or repeatability, for short) is measurement precision under a set of repeatability conditions of measurement.
2.20 a repeatability condition of measurement (repeatability condition) is a condition of measurement, out of a set of conditions that includes the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar objects over a short period of time.
2.25 measurement reproducibility (reproducibility) is measurement precision under reproducibility conditions of measurement.
2.24 a reproducibility condition of measurement (reproducibility condition) is a condition of measurement, out of a set of conditions that includes different locations, operators, measuring systems, etc. A specification should give the conditions changed and unchanged, to the extent practical.
The VIM definitions are squarely focused on measurement: repeatability and reproducibility are properties of measurements (not objects, scores, results or conclusions), and are defined as measurement precision, i.e. both are quantified by calculating the precision of a set of measured quantity values. Moreover, both concepts are defined relative to a set of conditions of measurement, in other words, the conditions have to be known and specified for assessment of either concept (repeatability, reproducibility) to be meaningful.

Mapping the VIM Definitions to a Metrological Framework for NLP/ML
The VIM definitions provide the definitional basis for reproducibility assessment. However, for a framework that tells us what to do in practice in an NLP/ML context, we also need (i) a method for computing precision, (ii) a specification of the practical steps in performing assessments, and (iii) a specific set of repeatability/reproducibility conditions of measurement. In this section, we present the fixed, discipline-independent elements of the proposed framework (Section 4.1), methods for computing precision in NLP/ML (Section 4.2), a series of steps for practical assessment of reproducibility (Section 4.3), and, utilising previous work on reproducibility in the NLP/ML field, starting points for identifying suitable conditions of measurement (Section 4.4). Table 2 gave a first indication what (some) conditions might look like in NLP/ML, grouped into Object conditions C O , Measurement method conditions C N , and Measurement procedure conditions C P . These conceptually useful groupings are retained to yield a skeleton framework shown in diagrammatic form in Figure 2, which corresponds to

Computing precision
Precision in metrological studies is reported in terms of some or all of the following: mean, standard deviation with 95% confidence intervals, coefficient of variation, percentage of measured quantity values within n standard deviations.
In reproducibility assessment in NLP/ML, sample sizes tend to be very small (a sample size of 8 as in Section 2.2 is currently unique). We therefore need to use de-biased sample estimators: we use the unbiased sample standard deviation, denoted s * , with confidence intervals calculated using a t-distribution, and standard error (of the unbiased sample standard deviation) approximated on the basis of the standard error of the unbiased sample variance se(s 2 ) as se s 2 (s * ) ≈ 1 2σ se(s 2 ) (Rao, 1973). Assuming measured quantity values are normally distributed, we calculate the standard error of the sample variance in the usual way: se(s 2 ) = 2σ 4 n−1 . Finally, we also use a small sample correction for the coefficient of variation: CV * = (1 + 1 4n )CV (Sokal and Rohlf, 1971).
Equipped with the above, the reproducibility of wF1 measurements of Vajjala and Rama (2018)'s MultPOS − system can, for example, be quantified based on the eight replicate measurements from Table 2 (disregarding the role of measurement conditions for the moment) as being CV * = 3.818. 7 This and two other example applications of the framework is presented in more detail in Section 5.

Steps in reproducibility assessment
In order to relate multiple repeatability and reproducibility assessments of the same object and measurand to each other and compare them, they need to share the same conditions of measurement (same in terms of attribute names, not necessarily values, see next section). Repeatability is simply the special case of reproducibility where all condition values are also the same.
For a true estimate of the variation resulting from given differences in condition values, the baseline variation, present when all condition values are the same, needs to be known. It is therefore desirable to carry out repeatability assessment prior to reproducibility assessment in order to be able to take into account the baseline amount of variation. E.g. if the coefficient of variation is x C 0 under identical conditions C 0 , and x C under varied conditions C, then it's the difference between x C 0 and x C that estimates the effect of varying the conditions. Finally, for very small sample sizes, both baseline repeatability assessment, and subsequent reproducibility assessment, should use the same sample size to ensure accurate assessment of variation due to the

REPEATABILITY PHASE
1. Select measurement to be assessed, identify shared object and measurand.
2. Select initial set of repeatability conditions of measurement C 0 and specify value for each condition.
5. Unless precision is as small as desired, identify additional conditions that had different values in some of the reproduction measurement, and add them to the set of measurement conditions, also updating the measurements to ensure same values for the new conditions. Repeat Steps 3-5.

REPRODUCIBILITY PHASE
6. From the final set of repeatability conditions, select the conditions to vary, and specify the different values to test. 7. For each combination of differing condition values: (a) Carry out n reproduction tests, yielding measured quantity values v 1 , v 2 , ...v n (b) Compute precision for v 1 , v 2 , ...v n , giving reproducibility score R.
Report all resulting R scores, alongside baseline R 0 score.

Figure 3:
Steps in 2-phase reproducibility assessment with baseline repeatability assessment.
Step 5 is obsolete if a field has standard conditions of measurement.
varied condition values (alone). Figure 3 translates what we have said in this and preceding sections into a 2-phase framework for reproducibility assessment. If a field develops shared standard conditions of measurement, Step 5 is obsolete. In situations where reproduction studies have been carried out without a pre-defined, shared set of conditions of measurement, reproducibility assessment can still be carried out (as we did for the Vajjala & Rama system in Section 2.2), but in this situation a baseline assessment of variation under repeatability conditions of measurement is not possible, and the only option is to use a single-phase version of the framework as shown in Figure 4. See Section 6 for more discussion of this issue.

Conditions of measurement
The final component needed for a metrological framework for reproducibility assessment in NLP/ML is a specific set of conditions of measurement. As mentioned in Section 4.1, individual conditions consist of name and value, and their role is to capture those attributes of a measurement where different values may cause differences in measured values. As indicated by Step 5 in Figure 3, for repeatability, conditions should be selected with a view to reducing baseline variation. As it's not practically feasible (or, normally, theoretically possible) to specify and control for all conditions that can be the same or different in a measurement, some judgment is called for here (see Section 6 for discussion). The idea is that a discipline evolves shared standard conditions that address this.
It so happens that much of the reproducibility work in ML and NLP has so far focused on what standard conditions of measurement (information about system, data, dependencies, computing environment, etc.) for metric measurements need to be specified in order to enable repeatability assessment, even if it hasn't been couched in these terms. Out of all reproducibility topics, pinning down the information and resources that need to be shared to enable others to obtain the same metric scores is the one that has attracted by far the most discussion and papers. In fact, reproduction in NLP/ML has become synonymous with rerunning code and obtaining the same metric scores again as were obtained in a previous study.
Reproducibility checklists such as those provided by (Pineau, 2020) and the ACL 8 are lists of types of information (attributes) for which authors are asked to provide information (values), and these can directly be construed as conditions of measurement. In this section, the intention is not to propose definitive sets of conditions relating to object, measurement method and measurement procedure that should be used in NLP/ML. Rather, in each subsection, we point to existing research such as the above that can be used to provide a 'starter set' of conditions of measurement.
2. Identify all conditions of measurement C for which information is available for all measurements, and specify values for each condition.
Report resulting R score. five items: specification of dependencies, training code, evaluation code, pre-trained models, README file including results and commands. These provide a good starting point for object conditions of measurement (note we group evaluation code under measurement method conditions, see next section): 1. Dependencies 2. Training code 3. Pre-trained models 4. Precise commands to run code/produce results 5. Compile-time environment 6. Run-time environment In human evaluation of system outputs, object conditions don't apply, as the object of measurement is fully specified by a sample of its outputs.

Measurement method conditions
In metric-based measurement where the measurand is defined mathematically, a measurement method is an implementation of a method for (or, conceivably, a manual way of) computing a quantity value for the measurand. In this case, the method, like the Object of measurement, is a computational artifact, so the same conditions can be used as in the preceding section (albeit with different names to mark the difference; values will clearly also differ).
In human-evaluation-based measurement, a different set of conditions is needed. Here, the measurand is identified by (ideally standardised) name and definition for the quantity being assessed. This has been termed 'quality criterion' 10 in previous work (Howcroft et al., 2020;Belz et al., 2020b) which took a first stab at a set of standardised names and definitions. These, in conjunction with a checklist for human evaluation such as the one proposed by , can provide a starting point for measurement method conditions (as follows), and measurement procedure conditions (next section), for human evaluation.
1. Name and definition of measurand 2. Evaluation mode 11 3. Method of response elicitation 4. Method for aggregating or otherwise processing raw participant responses 5. Any code used (conditions as for Object)

Measurement procedure conditions
Measurement procedure conditions capture any information needed to apply a given measurement method in practice. In metric-based measurement, this includes: 1. Test set 2. Any preparatory steps taken, such as preprocessing of text 3. Any code used (conditions as for Object) In human-evaluation-based measurement, some of the remaining properties from

Examples
In this section we apply the framework to the two sets of measurements from Sections 2.1 and 2.2, and an additional set of measurements from a recent reproduction study (Mille et al., 2021).

Mass of a Torc
In the case of the torc mass measurements, the reproducibility analysis is performed post hoc, i.e. we cannot obtain more information for the three older measurements than was recorded at the time.
We therefore have to use the 1-phase assessment (Figure 4), and for the complete set of seven weighings specify all conditions (Table 1) as different. CV * results can then be reported as follows: Mass measurement reproducibility under reproducibility conditions of measurement as detailed in Table 1, was assessed on the basis of seven measurements of torc 1991,0501.129 as follows: the unbiased coefficient of variation is 2.61, for a mean of 88.89, unbiased sample standard deviation of 2.24 with 95% CI (0.784, 3.696), and sample size 7. All measured values fall within two standard deviations, 71.43% within one standard deviation.
If we used just the four measurements for which all condition values are known, we would get CV * = 0.519 under reproducibility conditions where only the scales used differ. For a sample size of 4 we can still be reasonably confident that this is a good estimate of CV * for the whole population (stdev 95% CI [-0.04, 0.90]). Repeatability assessment can be performed for the two subsets of measurements for the SDS scales and the CBD scales, which gives CV * = 0.11 for the former and CV * = 0 for the latter, for similar-size stdev confidence intervals.

wF1 of a Text Classifier
The situation is similar for the text classifier wF1 measurements (Table 2) in that we are restricted to the information made available by the authors, which is however, more complete than in the torc example. Here too we have to use the 1-phase assessment (Figure 4), and for the complete set of 8 wF1 values specify all conditions as different. CV * results can then be summed up as above: wF1 measurement reproducibility under reproducibility conditions of measurement as in Table 2, was assessed on the basis of eight measurements reported by Vajjala and Rama (2018), Arhiliuc et al. (2020), Bestgen (2020), Caines and Buttery (2020), and Huber and Çöltekin (2020) (

Clarity and Fluency of an NLG system
The third example comes from a recent reproduction study (Mille et al., 2021) which repeated the human evaluation of a Dutch-language football report generation system (van der Lee et al., 2017). There were two main measurands, mean Clarity ratings and mean Fluency ratings, and a single object (the report generator) was evaluated. There were two scores for each of the measurands, one from the original study, one from the reproduction study. Here the situation was that a repeatability study was intended, but not possible. 12 Therefore a different set of evaluators and a different evaluation interface had to be used. Both Clarity and Fluency ratings were obtained on a 7-point scale (1..7). Computed on the scores as reported, for Clarity, CV * = 10.983, for Fluency, CV * = 13.525. However, if the 7-point scale had been 0..6, higher CV * values would have been obtained, i.e. results are not comparable across different rating scales. To address this, rating scales should be mapped to range with lowest score 0. Results can then be reported as follows: Clarity measurement reproducibility under reproducibility conditions of measurement as detailed in Mille et al. (2021)

Notes on Examples
CV * is fronted in the examples above as the 'headline' result. It can take on this role, because it is a general measure, not in the unit of the measurements (unlike mean and standard deviation), providing a quantification of precision (degree of reproducibility) that is comparable across studies (Ahmed, 1995, p. 57). This also holds for percentage within n standard deviations but the latter is a less recognised measure, and likely to be the less intuitive for many. CV is a measure of precision and as such, sample size should be ≥ 3. For a sample of 2 (as in the human evaluation in the last section), CV * is still meaningful as a measure of the variation found in the sample. However, it will generally provide a less reliable estimate of population CV.

Discussion
Specifying conditions of measurement in reproducibility assessment is important so results can be compared across different measures and assessments. We have not attempted to come up with a definitive set of conditions, but pointed to other research as a starting point. One important role the conditions play is to facilitate estimation of baseline variation via repeatability testing. It could be argued that if the goal is to ensure as near as possible identical measurements in repeatability testing, then a straightforward way to achieve that is containerisation. However, firstly the purpose of repeatability testing is to assess variation under normal use and it's not realistic to always run systems in Docker containers even in a research context. Secondly, human evaluation can't be run in a container. What counts as a good level of reproducibility can differ substantially between disciplines and contexts. E.g. in bio-science assays, 13 precision (reported as coefficient of variation) ranges from typically <10% for enzyme assays, to 20-50% for in vivo and cell-based assays, and >300% for virus titer assays (AAH/USFWS, n.d.). For NLP, such typical CV ranges would have to be established over time, but it seems clear that we would expect typical CV for metric-based measurements to be much lower (better) than for human-assessmentbased measurements. For a set of wF1 measurements, the 3.8% CV * above seems high.
There are many ways in which results from similar studies can be compared and conclusions drawn from comparisons. For example, to make (subjective) judgments of whether the same conclusions can be drawn from a set of comparable experimen-tal results, or to ask a group of assessors to make such judgments, is a valid and informative thing to do, but it's not reproducibility assessment in the general scientific sense of the term. It can also be informative to consider the ease with which systems can be recreated, but reproducibility is not a property of systems. Computer science has a history of departing from standard scientific reproducibility definitions, e.g. the ACM changed its definitions after NISO asked it to "harmonize its terminology and definitions with those used in the broader scientific research community." (Association for Computing Machinery, 2020). The question is, if the standard scientific terminology and definitions work for computer science, why would we not use them exactly as they are, rather than adapt them in often fundamental ways?

Conclusion
The reproducibility debate in NLP/ML has long been framed in terms of pinning down exactly what information we need to share so that others are guaranteed to get the same metric scores (e.g. Sonnenburg et al., 2007). What is becoming clear, however, is that no matter how much of our code, data and ancillary information we share, residual amounts of variation remain that are stubbornly resistant to being eliminated. A recent survey  found that just 14% of the 513 original/reproduction score pairs analysed were exactly the same. Judging the remainder simply 'not reproduced' would be of limited usefulness, as some are much closer to being the same than others, while assessments of whether the same conclusions can be drawn are prone to low levels of agreement. Quantifying closeness of results, and, over time, establishing expected levels closeness, seems a better way forward.
In this paper our aim has been to challenge the assumption that the general scientific reproducibility terms and definitions are not applicable or suitable for NLP/ML by directly mapping them to a practical framework that yields quantified assessments of degree of reproducibility that are comparable across different studies.
The NLP/ML field certainly needs some way of assessing degree of reproducibility that is comparable across studies, because the ability to assess reproducibility of results, hence the trustworthiness of evaluation practices, is a cornerstone of scientific research that the field has not yet fully achieved.

A Verbatim VIM Definitions
Primary term (synonyms) Definition 2.21 measurement repeatability (repeatability) measurement precision under a set of repeatability conditions of measurement 2.20 (Note 1) repeatability condition of measurement (repeatability condition) condition of measurement, out of a set of conditions that includes the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar objects over a short period of time NOTE 1 A condition of measurement is a repeatability condition only with respect to a specified set of repeatability conditions. 2.25 measurement reproducibility (reproducibility) measurement precision under reproducibility conditions of measurement 2.24 (Note 2) reproducibility condition of measurement (reproducibility condition) condition of measurement, out of a set of conditions that includes different locations, operators, measuring systems, and replicate measurements on the same or similar objects NOTE 2 A specification should give the conditions changed and unchanged, to the extent practical.

quantity
property of a phenomenon, body, or substance, where the property has a magnitude that can be expressed as a number and a reference 1.19 quantity value (value of a quantity, value) number and reference together expressing magnitude of a quantity 2.1 measurement process of experimentally obtaining one or more quantity values that can reasonably be attributed to a quantity 2.3 measurand quantity intended to be measured 2.5 measurement method method of measurement generic description of a logical organization of operations used in a measurement 2.6 measurement procedure detailed description of a measurement according to one or more measurement principles and to a given measurement method, based on a measurement model and including any calculation to obtain a measurement result NOTE 1 A measurement procedure is usually documented in sufficient detail to enable an operator to perform a measurement. 2.9 measurement result (result of measurement) set of quantity values being attributed to a measurand together with any other available relevant information 2.10 measured quantity value (value of a measured quantity, measured value) quantity value representing a measurement result 2.15 measurement precision (precision) closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions 4.1 indication quantity value provided by a measuring instrument or a measuring system Table 3: Verbatim VIM definitions of repeatability, reproducibility and related concepts (JCGM, 2012). (In VIM, definitions also give the earlier definition number from the second edition in parentheses which we omit here. In Definition 2.20, Note 2 relating only to chemistry is omitted. In 2.5 and 2.6, less important notes are omitted for space reasons.)