## QUANTUM PROGRAM TESTING THROUGH COMMUTING PAULI STRINGS ON IBM'S QUANTUM COMPUTERS

Asmar Muqeet Simula Research Laboratory University of Oslo Oslo asmar@simula.no

## Shaukat Ali

Simula Research Laboratory and Oslo Metropolitan University Oslo shaukat@simula.no

Paolo Arcaini National Institute of Informatics Tokyo arcaini@nii.ac.jp

## ABSTRACT

The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present QOPS , a novel quantum software testing approach. QOPS introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. QOPS also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making QOPS practical for testing complex quantum programs in industrial settings. We empirically evaluate QOPS on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of QOPS by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.

K eywords Software Testing, Test Oracle, Quantum Computing, Pauli Strings

## 1 Introduction

Quantum software testing aims to efficiently detect bugs in quantum software to ensure its correctness and reliability [Murillo et al.(2024)]. Existing quantum software testing methods adopt classical software testing methods such as search-based testing [Wang et al.(2022a), Wang et al.(2022b)], mutation testing [Mendiluze et al.(2021)], combinatorial testing [Wang et al.(2021a)], coverage criteria [Ali et al.(2021), Long and Zhao(2024), Ye et al.(2023)], propertybased testing [Pontolillo and Mousavi(2023)], mutation testing [Mendiluze et al.(2021), Fortunato et al.(2022)], metamorphic testing [Abreu et al.(2023)], and fuzzing [Wang et al.(2021c)]. However, practical limitations exist in applying these existing quantum software testing methods to real quantum computers.

First, the existing testing methods are incompatible with some types of quantum programs, specifically those performing search and optimization tasks. These quantum programs do not explicitly take inputs, but instead leverage the superposition of all states to find solutions to problems. In contrast, the types of quantum programs supported by existing testing methods require explicit inputs, and their testing involves finding failing inputs.

Second, most existing quantum testing methods rely on explicit oracles to determine whether a test case passes or fails. These methods often assume that the expected output for each test case is known in advance, which is unrealistic, especially for large quantum programs. As quantum algorithms evolve to handle more complex computations, specifying expected outputs for all potential test cases becomes increasingly impractical.

Third, existing testing methods assume ideal, noise-free quantum computers. However, this is not true for current quantum computers, which are inherently noisy, with errors affecting their computations [Resch and Karpuzcu(2021)]. To mitigate noise errors, the industry has adopted error mitigation methods that can be integrated with optimization programs to solve specific tasks on real quantum computers [LaRose et al.(2022)]. However, current testing methods are not designed to accommodate these error mitigation methods effectively. They often require additional instrumentation to align with error mitigation methods, making them difficult to use with high-level APIs such as IBM's Estimator API for developing quantum programs.

This paper addresses these practical challenges by introducing a novel quantum software testing approach called QOPS , which utilizes a new test case definition based on Pauli strings that does not require explicit inputs for quantum programs, thereby ensuring compatibility with all programs, including optimization and search programs. We also propose a new test oracle that can be integrated with error mitigation methods commonly adopted in the industry. Furthermore, we employ the concept of commuting Pauli strings and Pauli families to minimize the number of expected program outputs that must be specified to assess the passing and falling of a test case, making our approach more practical in industrial settings. We evaluated our approach on 194,982 real quantum programs and compared it with the current state-of-the-art test case definition and test oracle [Wang et al.(2021b)]. The results demonstrate that our approach performs better in test assessment, achieving an F1-score, precision, and recall of 1. Furthermore, we show the industrial relevance of our approach by performing test assessment on IBM's three quantum computers, integrating one industrial error mitigation method and one open-source method. Additionally, we discuss the implications and practicality of our approach and provide insights for practitioners.

Our key contributions are: 1) A novel testing approach that employs a new test case definition that is compatible with all quantum programs including search and optimization programs. 2) A novel test oracle compatible with error mitigation methods most widely used in industrial settings. 3) An empirical evaluation of test assessment across 194,982 quantum programs, demonstrating the effectiveness of our approach compared to an existing state-of-the-art. 4) An empirical evaluation of test assessment on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods to illustrate the practical application of our approach.

## 2 Background

## 2.1 QC Basics and Example

Qubits: In classic computing, the basic unit of information is the bit, which can only be in the state of 0 or 1. Quantum computing (QC), however, uses quantum bits or qubits , which can exist in a superposition of both | 0 ⟩ and | 1 ⟩ states, with associated amplitudes ( α ) . α is a complex number with both magnitude and phase in its polar form. A qubit is represented in Dirac notation [Dirac(1939)] as: | ψ ⟩ = α 0 | 0 ⟩ + α 1 | 1 ⟩ , where α 0 and α 1 are the amplitudes corresponding to states | 0 ⟩ and | 1 ⟩ , respectively. The probabilities of a qubit being in state | 0 ⟩ or | 1 ⟩ are determined by the square of the magnitudes of α 0 and α 1 , with the sum of these squared magnitudes equaling 1: | α 0 | 2 + | α 1 | 2 = 1 .

Quantum gates: These unitary operators change a qubit's state based on a unitary matrix [DiVincenzo(1998)]. For instance, the NOT gate flips the state of a qubit. Currently, gate-based quantum computers are programmed as quantum circuits, where the logic is implemented as a sequence of quantum gates applied to qubits. For better readability, we will refer to a quantum program as a 'circuit' throughout this paper. Figure 1 shows two entanglement circuits, each involving two qubits. Entanglement is a phenomenon where two or more qubits are linked in such a way that the state of one qubit directly affects the state of the other. The circuit starts with two qubits ( q 1 and q 2 ), initialized to the state | 0 ⟩ by default. First, a Hadamard gate ( H ) is applied to q 1 , placing it in a superposition of the | 0 ⟩ and | 1 ⟩ states. Next, a controlled-not gate ( ) is applied. A controlled-not gate is a two-qubit quantum gate, where the first qubit ( q 1 ) is referred to as the control qubit and the second qubit ( q 2 ) as the target qubit. The controlled-not gate flips the state of the target qubit if the control qubit is in state | 1 ⟩ . The controlled-not gate is applied after the Hadamard gate to create entanglement between the two qubits.

Measurement and Observable: Measurement is an operation used to extract the output of a quantum circuit. There are different types of measurement operations, with the type depending on the observable being used. Observables are physical quantities that can be measured, such as the position, momentum, and spin of a particle [Yanofsky and Mannucci(2008)]. In QC, observables are represented by Hermitian matrices [Yanofsky and Mannucci(2008)] with predefined eigenvalues, which are used in a measurement operation to extract the outcome of a qubit. Current quantum computers utilize four observables ( Z , Y , X I , ), known as Pauli observables , to measure the state of a single qubit [Yanofsky and Mannucci(2008)]. For multi-qubit circuits, the measurement operation is represented by the tensor product of single-qubit Pauli observables, referred to as Pauli strings . For instance, in Figure 1, the Z measurement ( Z ) in circuit 1 and the Y measurement ( Y ) in circuit 2 illustrate single-qubit Pauli observables. The overall measurement operation in circuit 1 is defined by Pauli string ZZ which is

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00501v1-figure-d6c6e572-06e6-4695-b17c-3a31e03991e7.jpg)

2. Entanglement Circuit With Y Measurement

Figure 1: Two-qubit entanglement quantum circuit with Z and Y measurement.

the tensor product of Pauli observable Z on qubit q 1 and q 2 . The output of a quantum circuit depends on the Pauli string used for measurement. For example, in Figure 1, the output for Pauli string ZZ in circuit 1 is (00) and (11), each with a probability of 0.5. For Pauli string YY in circuit 2, the output is (01) and (10), each with a probability of 0.5. For an n-qubit quantum circuit, there are 4 n possible combinations of Pauli strings, where n is the number of qubits and 4 represents the choices of Pauli observables ( Z Y , , X I , ) for each qubit. Pauli strings have eigenvalues associated with them, which provides information about the possible outcomes of a Pauli string [DiVincenzo(1998)]. As mentioned before, Pauli strings are tensor products of Pauli observables, and Pauli observables are Hermitian metrics with predefined eigenvalues. So the eigenvalues of Pauli strings are represented by taking the tensor product of eigenvalues of Pauli observables that makes up that Pauli string . For example Pauli observable Z has eigenvalues (1,-1). So the eigenvalues of Pauli string ZZ can be calculated by the tensor product of (1,-1) with (1,-1), resulting in (1, -1, -1, 1) as the eigenvalues of Pauli string ZZ . The eigenvalues play an important role in the calculation of expectation values (described next) of a Pauli string.

Expectation Value: The output of a quantum circuit can vary upon repeated measurements due to its inherent probabilistic nature [DiVincenzo(1998)]. The expectation value is a concept used to describe the average outcome one would anticipate from such repeated measurements. It can be calculated using the formula E = ∑ n i =0 w P i i , where n is the total number of possible output states of a circuit, P i is the probability of the i th output state, and w i is the i th eigenvalue of the Pauli string. For example, consider the two-qubit circuit in Figure 1 with Pauli string ZZ . The eigenvalues for ZZ are (1, -1, -1, 1). Using the formula and the measurement results from Figure 1, the expectation value for Pauli string ZZ is calculated as E = (1 ∗ 0 5) + ( . - ∗ 1 0) + ( - ∗ 1 0) + (1 ∗ 0 5) = 1 . .

## 2.2 Commuting Pauli Family

Pauli family refers to a group of Pauli strings that share a specific property, such as the commuting property [Reggio et al.(2023)], which we consider in this work. The commuting property states that the product of two Pauli strings is not dependent on the order of application. For example, two Pauli strings ZZ and ZI commute if ZZ ZI · = ZI · ZZ . As explained in section 2.1, for an n -qubit quantum circuit, there are 4 n Pauli strings, where n is the number of qubits and 4 arises from having four possibilities for each qubit, corresponding to the four types of Pauli observables: X Y , , Z , and I . All 4 n Pauli strings can be divided into K commuting Pauli families, where K varies for different quantum circuits. Several algorithms exist for generating Pauli families for a quantum circuit, with the state-of-the-art method presented by Reggio et al. [Reggio et al.(2023)]. One key advantage of constructing families of commuting Pauli strings is their ability to enable simultaneous measurements. Commuting Pauli strings have the property that the measurement result of a single Pauli string within a family can be used to calculate the expectation values of all Pauli strings within that family [Reggio et al.(2023), Kurita et al.(2023)]. This property of commuting Pauli strings is extensively utilized in current quantum computers for tasks such as error correction and accelerating quantum computations [Kurita et al.(2023)].

## 3 Industrial Context and Challenges

In this section, we present the industrial context by highlighting three main challenges associated with applying existing quantum software testing methods from the perspective of IBM's real quantum computers. These challenges are not exclusive to IBM, but also apply to other gate-based quantum computers. However, the observations presented here are from our experience in testing quantum circuits using IBM's quantum computers. The three main challenges are incompatible test cases , requirement of a complete program specification , and incompatibility with error mitigation methods .

## 3.1 Incompatible Test Cases

The first major issue with current quantum software testing methods is that they are not compatible with some types of quantum circuits. Quantum computing's primary application areas, such as physics simulations and quantum chemistry, revolve around optimization and search problems [Bayerstadler et al.(2021)]. The industry is particularly focused on developing quantum circuits that can function on today's noisy quantum computers to address various optimization and search problems. Examples of such circuits are the Quantum Approximate Optimization Algorithm (QAOA) and Grover's algorithm [Blekos et al.(2024), Bayerstadler et al.(2021)]. However, the existing quantum software testing methods do not support these optimization and search circuits. The major challenge existing testing methods face for search and optimization circuits is the definition of appropriate test cases. Currently, test cases for quantum circuits are defined in two main ways. The first is binary test cases , employed by several testing approaches [Wang et al.(2021b), Wang et al.(2022a), Wang et al.(2022b), Wang et al.(2021a), Muqeet et al.(2024c)]. In this definition, the test cases are represented as binary inputs in which qubits are initialized to either the | 0 ⟩ or | 1 ⟩ state. The second is quantum test cases , used by other testing methods [Pontolillo and Mousavi(2023), Wang et al.(2021c), Ye et al.(2023)]. Quantum test cases are represented as quantum inputs in which qubits are initialized to a random superposition state α | 0 + ⟩ β | 1 ⟩ . However, both these test case types are unsuitable for search and optimization circuits like Grover's and QAOA. Indeed, in these circuits, the qubits are initialized to specific superposition states depending on the problem to be solved. For example, in Grover's circuit, the qubits are initialized in an equal superposition of all states, which overwrites any specific initial state set by test cases. This makes traditional test cases unsuitable for testing such circuits.

It is therefore important to have a suitable test case definition for search and optimization circuits, especially when real quantum computers are predominantly used for quantum simulations and optimization tasks [Bayerstadler et al.(2021)]. We address this challenge by proposing a new test case definition in our approach consisting of Pauli strings that act on the measurement operation rather than explicitly setting qubit states as traditional test cases do.

## 3.2 Requirement of a Complete Program Specification

The second major issue is the impractical requirement of specifying the expected outputs for all test cases prior to executing the tests. Given that optimization circuits are intended to solve large-scale problems, it is impractical to define the output for all possible test cases. Current quantum software testing methods require explicitly stating whether a test case passes. This is done by comparing the observed test case outcome with the expected test case outcome defined in a program specification. However, creating a comprehensive program specification is challenging, particularly when quantum algorithms are designed to solve computationally intensive problems. For instance, if we only consider binary test cases, a quantum circuit with n qubits can have 2 n potential test cases, necessitating an overwhelming number of test cases to be specified. Although it may be feasible to specify the expected output of a few test cases, specifying the output of all test cases is computationally impractical. We reduce the cost of program specification by utilizing the commuting property of Pauli strings described in Section 2.2. Pauli strings can be grouped into commuting Pauli families, allowing one Pauli string from each family to determine the results for the others within that family [Reggio et al.(2023)]. By leveraging this property, the number of outcomes that need to be specified in the program specification can be significantly reduced (more details in Section 4).

## 3.3 Incompatibility with Error Mitigation

The third issue is the assumption made by most current testing methods that quantum computers are ideal and noisefree, which is not the reality. Indeed, real quantum computers exhibit inherent noise, which significantly affects both computational processes and the reliability of circuit outputs [Resch and Karpuzcu(2021)]. However, most current quantum software testing methods assume ideal conditions, making them ineffective in noisy real-world environments. Only three methods [Muqeet et al.(2024c), Muqeet et al.(2024a), Patel and Tiwari(2021)] enable testing on real quantum computers, but these methods come with the cost of training machine learning models, which requires a large

Figure 2: An example compact program specification ( PS compact ) for the entanglement circuit from Figure 1, showing the outputs of two Pauli strings ZX and YI ZX . belongs to a commuting family consisting of ( XY ZX YZ , , ) and YI belongs to a commuting family consisting of ( IY , YY YI , ).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00501v1-figure-ac275cee-10e1-4a88-b6b9-02941d862a74.jpg)

amount of data. An alternative approach for testing on real computers could involve using the error mitigation methods proposed in the literature [Cai et al.(2023)]. However, these error mitigation methods primarily focus on reducing noise effects on the expectation values of quantum circuits rather than individual outputs, whereas quantum testing methods operate with individual outputs and lack mechanisms or oracles directly addressing expectation values. This creates an incompatibility between current testing methods and error mitigation methods. Furthermore, companies such as IBM have introduced specialized APIs, such as the Estimator API in Qiskit [Javadi-Abhari et al.(2024)], to ease calculating expectation values with built-in error mitigation methods. However, current testing methods lack oracles directly addressing expectation values and are incompatible with such APIs. We address this challenge by proposing a new test oracle (Section 4.2.2) based on expectation values, making it compatible with IBM's Estimator API and other error mitigation methods.

## 4 Approach

We present an approach named QOPS : Q uantum pr O gram testing through P auli S trings, addressing the three main challenges outlined in Section 3. In QOPS , we utilize Pauli strings for testing quantum circuits. QOPS requires two inputs from a tester: the circuit under test (CUT) and a compact program specification ( PS compact ). In the PS compact , a tester is required to specify the expected output of any single Pauli string belonging to a commuting Pauli family. The cost of PS compact is significantly lower than the full program specification required by other testing methods described in Section 7. For example, for a 5-qubit circuit under test, the total number of Pauli strings is 4 5 (1024). Given that all 1024 Pauli strings can be partitioned into 33 commuting Pauli families, the PS compact would require a maximum of 33 Pauli strings, which is significantly lower than 1024. An example of PS compact is shown in Figure 2, which specifies the outputs of two Pauli strings ZX and YI for the entanglement circuit in Figure 1. In the next subsections, we first provide the definition of our test cases and the test oracle along with their rationale in Section 4.1. The overall working of the approach is described in Section 4.2.

## 4.1 Test Case and Oracle

QOPS employs a new definition of test case consisting of Pauli strings that act on the measurement operation rather than explicitly setting qubit states as traditional test cases do. We believe that different Pauli strings can identify the same faults in a circuit that are captured by explicitly setting quantum states. The rationale is that traditional test cases set the initial state of a circuit to observe a specific output. However, we noticed that varying the measurement operation with different Pauli strings can also produce the same output. In Figure 3, we show an intuitive example by considering the entanglement circuit as the circuit under test. Figure 3a shows the execution result of a binary test case 01 , which sets qubit q 1 to | 0 ⟩ and q 2 to | 1 ⟩ for the test circuit. However, we can also observe the same execution result of the binary test case by using Pauli string YY as shown in Figure 3a and not setting the qubit states explicitly. Figure 3b shows a quantum test case execution result where the quantum state Ψ is set in a superposition of | 00 ⟩ and | 10 ⟩ with 75% and 25% probability for each state, respectively. We can achieve the same output result as of the quantum test case, if we execute the quantum circuit twice, once with the Pauli string ZZ and second with the Pauli string YY and merge the two outputs by a weighted sum of both distributions, with 75% weight for ZZ and 25% weight for YY . This process of merging the results of different Pauli strings is commonly used in quantum optimization circuits [Blekos et al.(2024)]. This indicates that merging outputs of different Pauli strings with different weights could be analogous to quantum test cases without explicitly setting the state of qubits. However, the combination of Pauli strings that are analogous to a specific quantum input for a specific circuit could be different for different circuits. The advantage of using Pauli strings as tests over traditional test cases is that they are associated with the measurement operation, which is independent of the initialization logic of a circuit, making them suitable for optimization and search circuits.

Test case definition: We define our test cases based on the formulation T = ∑ S i =0 c i Ps i , where S is a subset of Pauli strings chosen randomly from a particular Pauli family, Ps i is the i th Pauli string in S , and c i is the weight assigned

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00501v1-figure-d132e586-5e57-456d-8bc8-7d61678a7495.jpg)

a)

Binary Test Case and Pauli String

b) Quantum Test Case and Pauli String

Figure 3: Example comparison of traditional test case definition (Binary, Quantum) with Pauli Strings.

to the i th Pauli string. For example, consider the subset S consisting of IZ , ZI , and ZZ . Then, a test case T can be written as T = c 1 IZ + c 2 ZI + c 3 ZZ . Here, c 1 , c 2 , and c 3 are the weights assigned to the Pauli strings to define the contribution of each Pauli string in the overall output. We use the weighted sum format to define our test case because we hypothesize that it is analogous to the traditional quantum test cases. For example, we show in Figure 3b, that the output obtained by executing a quantum test case Ψ can also be obtained by executing the Pauli strings ZZ and YY and performing a weighted sum of their outputs. Therefore, we can represent the same quantum test case Ψ in our Pauli string test case definition as T = 0 75 . ZZ +0 25 . Y Y .

Test Oracle: With our test case definition, we can randomly generate many test cases. However, we also need an oracle to determine whether the circuit under test (CUT) has failed a test case. For this, we define the oracle Expected Expectation ( Exp ), which is the total expectation value of a test case T after execution. This value is calculated by ∑ Ps i =0 c E i i , where Ps is the total number of Pauli strings in test case T , E i is the expectation value of the i th Pauli string in test case T , and c i is the weight associated with the i th Pauli string in test case T . Using the expectation value formula from Section 2.1, we can transform the equation as

$$E x p = \sum _ { i = 0 } ^ { P s } c _ { i } E _ { i } = \sum _ { i = 0 } ^ { P s } c _ { i } \sum _ { j = 0 } ^ { P s _ { i } } w _ { j } P _ { j }$$

where Ps i is the i th Pauli string in test case T , P j is the probability of the j th output state in the output of Pauli string Ps i , and w j is the j th eigenvalue of the i th Pauli string. Our test case consists of commuting Pauli strings, and, therefore, the output of a single Pauli string can be used to calculate the expectation value of other Pauli strings. By utilizing this concept, we can simplify Equation 1 as

$$E x p = \sum _ { ( i, j ) \in I } c _ { i } w _ { i, j } M _ { j }$$

where I = { ( i, j ) 0 | ≤ i ≤ Ps and 0 ≤ j ≤ Ps i } , Ps is the total number of Pauli strings in test case T Ps , i is the i th Pauli string in the test case T , c i is the weight of the i th Pauli string in test case T , w i,j is the j th eigenvalue of the i th Pauli string in test case T , and M j is the value of the j th output state of the Pauli string result given in the PS compact .

For example, consider the test case T = 0 3 . ZI +0 4 . IZ ; the output of Pauli string ZZ is specified in PS compact as (00, 01, 10, 11) with a probability of (0.5, 0.2, 0, 0.3) respectively. The eigenvalues of ZZ are (1,-1,-1, 1). Given that ZZ ZI , , and IZ belong to the same commuting family, we can calculate the total expectation value of test case T by utilizing the output of ZZ . Substituting the values in Equation 2:

$$\cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \\ & E x p = ( c _ { 0 } w _ { 0, 0 } M _ { 0 } + c _ { 0 } w _ { 0, 1 } M _ { 1 } + c _ { 0 } w _ { 0, 2 } M _ { 2 } + c _ { 0 } w _ { 0, 3 } M _ { 3 } ) \\ & \quad + ( c _ { 1 } w _ { 1, 0 } M _ { 0 } + c _ { 1 } w _ { 1, 1 } M _ { 1 } + c _ { 1 } w _ { 1, 2 } M _ { 2 } + c _ { 1 } w _ { 1, 3 } M _ { 3 } ) \\ & E x p = ( 0. 3 * 1 * 0. 5 + 0. 3 * - 1 * 0. 2 + 0. 3 * - 1 * 0 + 0. 3 * 1 * 0. 3 ) \\ & \quad + ( 0. 4 * 1 * 0. 5 + 0. 4 * - 1 * 0. 2 + 0. 4 * - 1 * 0 + 0. 4 * 1 * 0. 3 ) = 1. 5 6 \\ \text{ad association value of that case $T$ would be $1$ $\xi$}$$

The expected expectation value of the test case T would be 1.56.

Figure 4: Overview of QOPS : The tuple ( F EV M , , ) includes the Pauli family ( F ), the eigenvalues ( EV ) of the Pauli strings in family F , and the Specified outcome ( M ) of a single Pauli string in F based on a compact program specification ( PS compact ).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00501v1-figure-c96f02a5-4f52-4130-a829-034866b163c4.jpg)

Table 1: Pauli Families with corresponding eigenvalues identified in Pauli Transformation stage for Entanglement circuit in Figure 1 taken as CUT.

| Family      | EigenValues                                                   |
|-------------|---------------------------------------------------------------|
| II,IZ,ZI,ZZ | (1, 1, 1, 1), (1, -1, 1, -1), (1, 1, -1, -1) , (1, -1, -1, 1) |
| IX,XI,XX    | (1, -1, 1, -1), (1, 1, -1, -1), (1, -1, -1, 1)                |
| IY,YI,YY    | (-1, 1, -1, 1), (-1, -1, 1, 1), (1, -1, -1, 1)                |
| XY,ZX,YZ    | (-1, 1, -1, 1), (-1, -1, 1, 1), (-1, 1, 1, -1)                |
| XZ,YX,ZY    | (-1, 1, -1, 1), (-1, -1, 1, 1), (-1, 1, 1, -1)                |

## 4.2 Overview

Figure 4 shows the overview of QOPS approach consisting of two main parts, Pauli Transformation , and Test Assessment , which are further described in the following subsections.

## 4.2.1 Pauli Transformation

QOPS first generates families of commuting Pauli strings that can be used for testing a quantum circuit by taking CUT and PS compact as inputs. To create Pauli families, QOPS employs the partitioning algorithm developed by Reggio et al. [Reggio et al.(2023)], details of which can be found in their respective work. Here, we summarize the partitioning algorithm's operation in two main steps. First, the algorithm starts by identifying a set of Pauli strings that fulfills the commuting condition. Next, for a given CUT and the selected set of Pauli strings, it finds a matrix that diagonalizes all the Pauli strings in the selected set. If such a matrix is successfully found, the commuting set is considered one family, and the process is repeated until no more sets can be found. The output of the partitioning algorithm is a list of K Pauli families and the eigenvalues of each Pauli string within a family. Table 1 shows the output of the partitioning algorithm for the entanglement circuit from Figure 1 taken as CUT. The column Family shows the identified Pauli Families and the column EigenValues shows the eigenvalues associated with each Pauli string within a Pauli family.

```
1. Testcase = 0.75 * SparsePauliOp( "ZZ" ) + 0.25 * SparsePauliOp( "ZI" )
2. service = QiskitRuntimeService()
3. backend = service.least_busy( operational=True, simulator=False )
4. IBM_estimator = Estimator( backend, options= { "resilience_level":  2 } )
5. job = IBM_estimator.run( CUT, Testcase )
6. obs = job.result().data.evs
```

Figure 5: An example of IBM Estimator API to execute the test case with inbuilt error mitigation enabled.

Given the measurement result of any Pauli string in a family, these eigenvalues can be utilized to calculate the test oracle for a test case consisting of any subset of Pauli string within the family as shown in Section 4.1.

Once QOPS has identified the Pauli families and the corresponding eigenvalues, it checks the PS compact to filter out non-relevant families for testing. Depending on the number of Pauli strings specified in the PS compact , not all families may be suitable for inclusion in the testing process. Ideally, the PS compact should specify at least one Pauli string for each Pauli family to ensure all families can be effectively utilized during testing. However, depending on the number of Pauli strings that can be specified by the tester, the number of applicable Pauli families are reduced accordingly. As shown in Figure 4, the result of the Pauli Transformation stage in QOPS is an applicable set of families. Each item in this set is represented as a tuple containing three values: F - a specific Pauli family, EV - the corresponding eigenvalues of the family, and M - the measurement outcome specified in PS compact for any Pauli string from the family F .

## 4.2.2 Test Generation and Assessment

The second part of QOPS aims to test the quantum circuit by utilizing the applicable family set from the Pauli Transformation stage. As shown in Figure 4, for each family in the applicable family set, QOPS generates random test cases along with their expected outcome for a specific budget. The user defines the budget, and decides how many test cases to generate for each family. In our experiments, we set the budget to 1000. As shown in Figure 4, the test assessment follows a simple process. For each family, a number of random test cases are generated depending on the budget, and the oracle for each test case is automatically calculated using the PS compact . The test cases and the CUT are then passed to IBM's Estimator API to execute the CUT with the test case and obtain the results. In case of error mitigation enabled, the result is first parsed to the error mitigation methods integrated in IBM's Estimator API or any open source error mitigation method. The observed results are then compared with the oracle using a threshold value set by the tester. If the difference between the observed result and the oracle exceeds the threshold, the test case is considered failed, and the fault is reported to the tester.

## 4.2.3 Integration with Error Mitigation

QOPS 's test case definition and oracle are directly compatible with error mitigation methods as we operate with the expectation value and not with individual outputs. Figure 5 illustrates an example of test case executed with IBM's Estimator API, using the zero noise extrapolation (ZNE) technique for error mitigation. In Line 1, we define a test case as T = 0 75 . ∗ ZZ +0 25 . ∗ ZI . In Lines 2-3, we select an IBM quantum computer that is least busy at the moment. Line 4 invokes the IBM Estimator API with the resilience level set to 2, enabling the inbuilt ZNE error mitigation technique. Finally, in Line 5, we submit our test case along with the CUT to the IBM quantum computer. Note that our test case definition is directly compatible, so we can pass the defined test case variable directly to the Estimator API. In Line 6, we obtain the error-mitigated observed expectation value of the test case, which we then compare with our own calculated expected expectation value for assertion.

## 5 Evaluation and Analysis

## 5.1 Research Question

We assess the effectiveness of QOPS in detecting faults in quantum circuits by answering the following research questions (RQs):

- RQ1 How comparable is QOPS to the most widely used test case definition in terms of capturing faults in quantum circuits?

RQ2 How effective is QOPS when applied to testing quantum circuits on real noisy quantum computers?

## 5.2 Experiment Design

## 5.2.1 Benchmarks

We employ the MQT benchmark [Quetschlich et al.(2023)], a repository of diverse quantum circuits with real-world applications for evaluation of QOPS . MQT includes 18 different quantum algorithms implemented with varying qubit sizes. We selected all 18 algorithms, but limited the number of qubits to between 2 and 10 to ensure reasonable execution times for the quantum simulator and real computers. This selection resulted in 157 quantum circuits for testing. Additionally, we created faulty versions of these 157 circuits using Muskit [Mendiluze et al.(2021)], applying all possible mutation operators defined in Muskit. This resulted in 194,982 quantum circuits, comprising 30,084 equivalent circuits and 164,898 faulty circuits, which were used to evaluate QOPS . We benchmarked QOPS against the most widely used test case definition, the binary test cases, and the common test oracle based on the chi-square statistical test. These are prevalent in current testing methods [Wang et al.(2021b), Wang et al.(2022a), Wang et al.(2022b), Wang et al.(2021a), Muqeet et al.(2024c)]. For better readability, we abbreviate the existing test case definition and oracle as ETO for the rest of the paper.

## 5.2.2 Experiment Settings

The implementation of QOPS and the experiment results are publicly available in our repository [Muqeet et al.(2024b)]. We use the Qiskit framework for quantum circuit execution [Javadi-Abhari et al.(2024)]. To automatically obtain the full program specifications for the benchmarks for ETO , we use Qiskit's AER simulator to get the correct circuit outputs. We also use Qiskit's AER simulator to get compact program specifications PS compact for QOPS , specifying only one Pauli string for each commuting family for a specific quantum circuit. For execution on real quantum computers, we use IBM's three publicly accessible quantum computers (IBM-Brisbane, IBM-Osaka, and IBM-Kyoto) and employ IBM's Estimator API for circuit execution. For the number of test cases, we set QOPS budget to 1000 test cases per Pauli family, whereas for ETO we used all possible test cases, which results in 2 n test cases, where n is the number of qubits in the circuit under test.

Regarding error mitigation on real computers, the most common methods are probabilistic error cancellation (PEC), zero-noise extrapolation (ZNE), and Clifford data regression (CDR) [LaRose et al.(2022)]. IBM's Estimator API provides built-in error mitigation based on ZNE and PEC. Given the exponential cost of PEC [Ma and Kim(2024)] and IBM's monthly limit of 10 minutes for using real quantum computers, we selected the ZNE method as the error mitigation method in the Estimator API. Additionally, we used an open-source version of the CDR method [LaRose et al.(2022)] for error mitigation to demonstrate QOPS 's applicability with both industrial and opensource error mitigation methods.

QOPS requires a tolerance threshold to assert whether a test case fails. For a fair comparison between QOPS and ETO , we set this threshold to the same value used in the statistical test assertion in ETO , which is 1% for evaluation on the quantum simulator. For real quantum computers, considering that error mitigation methods have their own error margins, we used the Standard Error [Curran-Everett(2008)] as the threshold. To calculate the standard error for a quantum circuit, we executed the non-faulty version 10 times with error mitigation and calculated the difference between the ideal value and the error-mitigated value for each run. The standard error was then calculated using the formula SE = σ √ n , where σ is the standard deviation of the differences between the ideal and error-mitigated values over 10 runs, and n is the total number of runs, which is 10. The standard error value varies depending on the margin of error of an error mitigation method. Thus, it is different for each quantum circuit executed with a specific error mitigation method.

## 5.2.3 Metrics

For RQ1, we compare the test assessment results of QOPS with those of ETO using a classification problem framework. In this context, a method's accuracy in distinguishing between faulty circuits and equivalent or non-faulty circuits indicates its effectiveness. Therefore, we compare QOPS with ETO based on commonly used quality metrics: F1-score, precision, and recall [Murphy(2007)]. These metrics are calculated using their standard formulas [Murphy(2007)]. We compute F1-score, precision, and recall separately for equivalent and faulty versions of quantum circuits, comparing the results with the ground truth for both QOPS and ETO . The ground truth is calculated based on the Hellinger distance [Dasgupta and Humble(2022)] between the original circuit and the mutant circuit generated by Muskit. Hellinger distance is a commonly used metric for checking the equivalence of two quantum circuits based on their outputs [Dasgupta and Humble(2022)]. All circuits that result in a zero value for Hellinger distance are classified as equivalent circuits, and others are classified as faulty circuits. For quality metric calculation, we define True Positive (TP), False Negative (FN), False Positive (FP), and True Negative (TN), in comparison to ground truth as follows: TP - Both the ground truth and ETO 's ( QOPS 's) test assessment say 'faulty'; FP -ETO 's ( QOPS 's) test assessment says

Table 2: RQ1 - Quality metric comparison between ETO and QOPS against the ground truth. The total number of circuits evaluated in the EqBenchs category is 30,084, and in the FaultyBenchs category is 164,898.

| Metric    | ETO      | ETO          | QOPS     | QOPS         |
|-----------|----------|--------------|----------|--------------|
| Metric    | EqBenchs | FaultyBenchs | EqBenchs | FaultyBenchs |
| F1-Score  | 0.992    | 0.999        | 1.000    | 1.000        |
| Precision | 0.984    | 1.000        | 1.000    | 1.000        |
| Recall    | 1.000    | 0.997        | 1.000    | 1.000        |

'faulty', but the ground truth says 'not faulty'; FN -ETO 's ( QOPS 's) test assessment says 'not faulty', but the ground truth says 'faulty'; and TN - Both the ground truth and ETO 's ( QOPS 's) test assessment say 'not faulty'.

For RQ2, since ETO is incompatible with error mitigation methods, we can only evaluate QOPS on real quantum computers. To demonstrate QOPS 's applicability on real computers, we use the same quality metrics as in RQ1. We then analyze how the results differ from our findings in RQ1 when QOPS is integrated with error mitigation methods and evaluated on real quantum computers.

## 5.3 Results

## 5.3.1 RQ1-Comparison with ETO

Table 2 compares ETO with QOPS in terms of F1-score, precision, and recall. The columns labeled EqBenchs and FaultyBenchs present the test assessment results for equivalent and faulty circuits for both ETO and QOPS when compared against the ground truth. Table 2 shows that QOPS outperforms ETO in accurately identifying faulty and equivalent circuits without any false positives or false negatives. Although ETO generally performed well, it exhibited higher false positives and false negatives for some equivalent and faulty circuits compared to QOPS . Specifically, ETO misclassified five out of 30,084 equivalent circuits, likely due to the chi-square statistical test's inherent margin of error for false positives. Regarding faulty circuits, ETO failed to detect faults in 484 out of 164,898 cases. These undetected faults predominantly involved a specific type of fault known as a phase flip fault [Oldfield et al.(2024)]. Detecting phase flip faults is particularly challenging, because it requires altering the Pauli observables in the measurement operation. The chi-square test employed by ETO is less effective in identifying this type of fault, leading to a higher number of undetected cases. On the other hand, QOPS proves to be highly effective in capturing phase flip faults. This effectiveness comes from the fact that QOPS utilizes test cases that consist of Pauli strings, where each Pauli string is composed of different Pauli observables. Overall, Table 2 shows that QOPS provides better accuracy in terms of identifying faulty circuits and equivalent circuits with no false positives and false negatives.

RQ1: Compared to state-of-the-art, QOPS is more effective in test assessment with a perfect F1-score, precision, and recall, indicating that QOPS is a more reliable and alternative approach for testing quantum circuits.

## 5.3.2 RQ2-Integration with error mitigation

RQ2assesses whether QOPS can be integrated with error mitigation methods and successfully capture faults in quantum circuits on real quantum computers. Given that IBM's quantum computers have a monthly usage limit of 10 minutes, executing all 194,982 circuits on real computers was not possible. To fit within these usage limits, we reduced the number of circuits from RQ1. We employed the Fisher-Jenks natural break algorithm [Fisher(1958)] to identify representative faulty circuits for the 157 original circuits, based on the diversity in the QOPS oracle results. The FisherJenks algorithm is a widely used method for automatically selecting a representative subset from large datasets based on diversity criteria [North(2009)]. The diversity criterion was the expected expectation value of the failed test cases from the RQ1 experiments. This algorithm resulted in 10 faulty circuits per original circuit, yielding 1,570 circuits to be evaluated on real quantum computers. For error mitigation methods, as stated in Section 5.2.2, we used both ZNE and CDR methods. IBM provides four different configurations of the ZNE method: Linear (L), Cubic (C), Quartic (Q), and Richardson (R), and we treated each configuration as a separate method. This resulted in five different error mitigation methods, which we evaluated on IBM's three real quantum computers.

Table 3 shows the results of evaluating 1,570 faulty circuits on the IBM's three real quantum computers: IBMBrisbane, IBM-Osaka, and IBM-Kyoto. The columns display the F1 score, precision, and recall for each error mitigation method and computer pair, calculated by comparing the results with the ground truth. From Table 3, we observe that QOPS performed well across all computer and error mitigation method pairs. The variation in F1 scores for each pair is attributed to the differing levels of noise in each computer and the varying effectiveness of error mitigation

Table 3: RQ2 - Quality metric comparison between different error mitigation methods integrated with QOPS against the ground truth evaluated on IBM's real quantum computers. ZNE configurations are also presented with initials as L-Linear, C-Cubic, Q-Quartic, and R-Richardson.

|                 | IBM-Brisbane   | IBM-Brisbane   | IBM-Brisbane   | IBM-Brisbane   | IBM-Brisbane   | IBM-Osaka   | IBM-Osaka   | IBM-Osaka   | IBM-Osaka   | IBM-Osaka   | IBM-Kyoto   | IBM-Kyoto   | IBM-Kyoto   | IBM-Kyoto   | IBM-Kyoto   |
|-----------------|----------------|----------------|----------------|----------------|----------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Metric          | CDR            | ZNE L          | ZNE C          | ZNE Q          | ZNE R          | CDR         | ZNE L       | ZNE C       | ZNE Q       | ZNE R       | CDR         | ZNE L       | ZNE C       | ZNE Q       | ZNE R       |
| Precision       | 1.000          | 1.000          | 1.000          | 1.000          | 1.000          | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       | 1.000       |
| Recall          | 0.986          | 0.999          | 0.996          | 0.985          | 0.998          | 0.983       | 0.998       | 0.993       | 0.986       | 0.999       | 0.982       | 0.999       | 0.992       | 0.985       | 0.998       |
| F1 score        | 0.993          | 1.000          | 0.998          | 0.993          | 0.999          | 0.991       | 0.999       | 0.997       | 0.993       | 1.000       | 0.991       | 1.000       | 0.996       | 0.992       | 0.999       |
| False Negatives | 22.00          | 1.000          | 5.000          | 15.00          | 2.000          | 24.00       | 2.000       | 8.000       | 14.00       | 1.000       | 27.00       | 1.000       | 10.00       | 16.00       | 3.000       |

methods for different circuits. Notably, the best error mitigation method integrated with QOPS is the ZNE linear configuration ( ZNE L ), which had only one false negative on IBM-Bisbane and IBM-Kyoto and two false negatives on IBM-Osaka. The second best method is the ZNE Richardson ( ZNE R ) method. Conversely, the worst method is CDR, with over 20 false negatives across all computers.

Each error mitigation method has its own time cost for filtering noise from the circuit output. Thus, choosing an appropriate error mitigation method is important. Table 3 indicates that IBM's integrated ZNE method outperforms the CDR method, particularly with its linear configuration. Overall, QOPS can successfully perform test assessments on real computers when integrated with error mitigation methods. Testers using IBM's APIs can conduct testing activities on real computers without additional instrumentation for integrating error mitigation with QOPS . However, the performance of QOPS is dependent on the accuracy of the error mitigation methods.

RQ2: Compared to state-of-the-art, QOPS is directly compatible with error mitigation methods, allowing QOPS to successfully perform test assessments on real computers using APIs with integrated error mitigation methods such as IBM's Estimator API.

## 5.4 Threats to validity

Construct Validity [Wohlin et al.(2012)] refers to how well a measurement evaluates the intended theoretical concept. One threat to construct validity involves the metrics used to assess the effectiveness of QOPS . A testing method must accurately differentiate between faulty and non-faulty circuits. The effectiveness of one testing method compared to another depends on the number of false positives and false negatives produced during the assessments. This work uses F1-score, precision, and recall as performance metrics. Low precision and recall indicate higher false positives and negatives, while the F1-score measures the overall performance of a test assessment method. Another concern is the selection of error mitigation methods and the real computers used. Different quantum computers exhibit varying noise levels, and different error mitigation methods have different error margins for different circuits executed on different computers. In this paper, we focused on all of IBM's quantum computers that were publicly available during the experiment. We concentrated on error mitigation methods commonly used in the industry. IBM provides two error mitigation methods: ZNE and PEC. Given our limited access to IBM's quantum computers, the only feasible error mitigation method was ZNE. Additionally, we used an open-source error mitigation method, CDR, to demonstrate the applicability of QOPS with both industrial and open-source error mitigation methods.

Internal Validity [Wohlin et al.(2012)] concerns how experiments can establish a causal relationship between independent and dependent variables. One threat to internal validity involves the tolerance parameter of QOPS . The tolerance parameter determines the acceptable error margin when comparing the observed expectation value of the test cases with the expected expectation. This parameter depends on the circuit and requires domain knowledge to be set correctly. In our experiments, to ensure a fair comparison with the ETO and QOPS , we chose the tolerance value to be the same as the threshold used in ETO 's statistical test for passing and failing a hypothesis, which is 1%. Although using a different tolerance value could affect the outcome, we found in the RQ1 results that a tolerance value of 1% was optimal, as it resulted in zero false positives and negatives for QOPS . For evaluations on real computers, since error mitigation methods have their own error margins that vary for different circuits, a single tolerance value cannot be established. To address this, we used 1% plus the standard error value of the original circuit for 10 consecutive runs executed with error mitigation as the threshold value. The standard error is calculated automatically for each circuit and error mitigation pair before performing the full test assessment.

Conclusion validity [Wohlin et al.(2012)] focuses on the statistical significance of the results derived from an experiment. One threat to conclusion validity is using random test cases in the test assessment by QOPS . Using random test cases does not guarantee that QOPS will perform consistently if executed multiple times. However, the main objective of this study is to assess whether QOPS can identify all the faults. Therefore, we focused on evaluating a large number

```
                                                                                                                                                                                                        QSlkit                                                                                                                                                                                                       
                                                                                                                                                                                                       	                                                                                                                                                                                                        ZZ = SparsePauliOp("ZZ")                                                                                                                                                                                                        zZ = PauliString(cirq.Z(0), cirq.Z(0))                                                                                                                                                                                                        xZ = PauliWord({0:"Z", 1:"Z"})                                                                                                                                                                                                        \
                                                                                                                                                                                                      
	                                                                                                                                                                                                      
Testcase = 0.75*ZZ+0.25*Z1                                                                                                                                                                                                        \\
                                                                                                                                                                                                       \                                                                                                                                                                                                        }
                                                                                                                                                                                                       }                                                                                                                                                                                                        0
                                                                                                                                                                                                       0                                                                                                                                                                                                        1
                                                                                                                                                                                                      		                                                                                                                                                                                                       0
```

Figure 6: An example of a test case in three widely used quantum software stacks: IBM's Qiskit, PennyLane's QML, and Google's Cirq.

of circuits, which limited our ability to run the experiment multiple times. Additionally, we set the number of test cases for each Pauli family to 1000 to increase the probability of finding a test case that captures the fault in a faulty circuit. The RQ1 results indicate that QOPS , even with random test cases, could correctly identify all the faulty circuits. Multiple runs to reduce random bias are only necessary when there is a chance that a better solution exists. However, in our case, a single experiment run was sufficient to find the best solution, reducing the need for multiple experiment evaluations.

External Validity [Wohlin et al.(2012)] refers to how our method can be applied to other datasets and domains. A challenge in this area is the selection of quantum circuits for the test assessment, as a diverse set of circuits would result in more generalizable outcomes. To mitigate this issue, we used the widely accepted benchmark MQT [Quetschlich et al.(2023)] to choose circuits representing the most commonly utilized quantum circuits. Another potential threat is the number of qubits used, as increasing the number of qubits directly adds to the complexity of quantum circuits. Current quantum simulators can only handle up to 10 qubits in a reasonable time [Jaeger(2007)]. In this study, our goal was to test many circuits for generalizability, which restricted the number of qubits we could use, as more qubits significantly increase the execution time of circuits on simulators. Thus, to evaluate a large number of circuits within a practical time frame, we set the maximum number of qubits to 10.

## 6 Discussion

## 6.1 Practical Implications

This study presented a new quantum circuit testing approach ( QOPS ) to solve three practical challenges to ensure the reliable development of QC solutions in an industrial context, particularly for IBM's real quantum computers. Indeed, the incompatibility of current quantum software testing methods with circuits designed for optimization and search tasks limits their practical utility. Since the primary industrial application of QC is in optimization and search problems, this gap presents a substantial barrier. Additionally, the impractical requirement to specify expected outputs for all test cases before execution, challenges the scalability and practicality of current testing methods in real-world scenarios. Furthermore, the assumption that quantum computers are ideal and noise-free fails to reflect their actual state. In reality, quantum computers are susceptible to noise, which affects their performance and accuracy.

To address these practical challenges, especially for testers using IBM's APIs for testing, we presented a new definition of test cases and a test oracle in QOPS . The QOPS 's generated test cases are directly compatible with IBM APIs, such as the Estimator API, which is heavily used for optimization and search circuits. We utilize the commuting property of Pauli strings to reduce the need for large number of predefined expected outputs and provide a new test oracle compatible with error mitigation methods, particularly those implemented in IBM APIs. Our empirical evaluation shows that QOPS can be a potential alternative for existing quantum circuit testing methods, improving the reliability and scalability of quantum computing methods developed using IBM's quantum computers.

## 6.2 Integration with Other Quantum Software Frameworks

Our approach demonstrates that the test case definition and oracle of QOPS are fully compatible with IBM's Qiskit framework. While our current implementation primarily supports IBM's API for automated test assessment, our test case definition and oracle are also compatible with other widely used gate-based quantum computers and software frameworks. These include IBM's Qiskit, PennyLane's QML, and Google's Cirq [Hidary and Hidary(2021)]. Although the execution of quantum circuits varies among these frameworks, the fundamental definition of the test case and oracle remains consistent. For instance, consider a test case T = 0 75 . ∗ ZZ +0 25 . ∗ ZI , comprising two Pauli strings: ZZ and ZI . Figure 6 illustrates how this test case can be represented in the three most popular frameworks: Qiskit, QML, and Cirq. All three frameworks support the concept of Pauli observables, enabling straightforward translation of the mathematical representation of test case T into an executable form. In Qiskit, the test case can be instantiated using the SparsePauliOp object; in QML, using the PauliWord and PauliSentence objects; and in Cirq, using the PauliString and PauliSum objects. Each framework provides an API that accepts the test case and the cir-

cuit under evaluation, returning expectation values accordingly. Regarding QOPS 's test oracle, no transformations are necessary for different frameworks, because the oracle is computed directly from the mathematical representation of the test case. Therefore, quantum software developers using various quantum software stacks can seamlessly integrate QOPS into their development environments.

## 6.3 Scalability

In terms of test assessment for quantum circuits, QOPS is a highly scalable method compared to existing testing methods. QOPS is capable of generating and evaluating K × N test cases from just K Pauli strings specified in PS compact , where K represents the number of commuting Pauli families for a quantum circuit, and N denotes the number of random test cases. Existing testing methods, which require a full program specification to determine the passing and failing of test cases, are limited to evaluating only the number of test cases explicitly specified in the program specification. In contrast, QOPS leverages the specified Pauli strings in PS compact to generate as many test cases as the test budget allows, making it a practical approach for scenarios where the program specification is limited.

## 6.4 Way Forward

In this study, we introduced an alternative approach for quantum circuit testing by developing a new definition for test cases and a new test oracle. We demonstrated that quantum circuit faults can be effectively identified through randomly generated test cases. However, generating random test cases is time-consuming and resource-intensive, making it a highly inefficient method for testing.

To enhance the efficiency and effectiveness of our method, we plan to improve the test case generation aspect of QOPS in the future. One promising solution is to employ search-based approaches such as genetic algorithms, simulated annealing, or particle swarm optimization [Harman et al.(2012)]. These algorithms can systematically explore the space of potential test cases, prioritizing those that are more likely to uncover faults efficiently. By using these search methods, we hope to significantly reduce the time and resources required for test case generation.

Another potential research direction is to discover relationships between test cases and various types of faults. Specifically, focusing on the relation between the passing or failing of tests within the same Pauli family. For instance, if one test in a family fails, is it more likely that other tests in the same family will also fail? Moreover, even if a test does not fail, could its execution provide a hint that another test in the same family might fail? Understanding the relation between specific test cases and the faults they reveal could lead to new methods for debugging and repairing quantum circuits. This knowledge would enhance the accuracy of fault detection and provide valuable insights for developing more robust quantum circuits.

## 7 Related Work

Several studies have explored quantum software testing, introducing various methods for testing quantum circuits, evaluating test suite quality, and providing test oracles. Various tools and approaches have been proposed in the literature for testing quantum circuits. Quito [Wang et al.(2021b)] is a tool proposed to test quantum circuits by fulfilling input-output coverage criteria. Tools like Muskit [Mendiluze et al.(2021)] and QMutpy [Fortunato et al.(2022)] assist in mutation testing. QuSBT [Wang et al.(2022a)] uses a search algorithm to generate test cases and MuTG [Mendiluze et al.(2021)] us a multiobjective search algorithm to find a minimum number of test cases that can kill a maximum number of mutants. QuCAT [Wang et al.(2021a)] employs combinatorial testing methods, while QuanFuzz [Wang et al.(2021c)] utilizes fuzzing method for quantum circuit testing. Several methods have been developed to gauge the effectiveness of test suites. Coverage criteria such as input-output coverage [Ali et al.(2021)] and equivalence class partitioning [Long and Zhao(2024)] are employed to evaluate test suite effectiveness. Currently, two common types of oracles are used for quantum circuit testing. The first type employs statistical tests to compare quantum circuit outputs [Ali et al.(2021), Huang and Martonosi(2019)]. The second type, property-based testing [Honarvar et al.(2020), Pontolillo and Mousavi(2023)], derives several properties for the circuit under test, where the violation of these properties indicates the presence of faults. Several bug identification frameworks [Luo et al.(2022), Zhao et al.(2021a), Miranskyy et al.(2020)] have also been proposed, including Bugs4Q [Zhao et al.(2021b)], which is a benchmark for real-world Qiskit circuit bugs.

Current real-world quantum computers are noisy, which greatly reduces the reliability of computations and introduces errors in circuit outputs [Resch and Karpuzcu(2021)]. The causes of noise in quantum computers are environmental factors such as magnetic fields and radiation, qubit cross interactions, and calibration error in quantum gates [Georgopoulos et al.(2021)]. Due to noise, testing quantum circuits on actual quantum computers is a challenging task because it becomes difficult to differentiate between faults and errors from noise. Several methods have been

proposed for mitigating quantum noise [Cai et al.(2023)], focusing on reducing noise effects on the expectation values of quantum circuits. To the best of our knowledge, only three testing methods enable quantum circuit testing on real computers. The first method is QOIN [Muqeet et al.(2024c)] which trains a machine-learning (ML) model utilizing three statistical features to reduce the effect of noise. The second is Qraft [Patel and Tiwari(2021)], which introduced a new set of features for training ML models that can be applied to multiple circuits under test. The last one is QLEAR [Muqeet et al.(2024a)], which introduced a new feature called depth-cut-error to address Qraft's limitation of overestimating noise in the circuit output.

The main drawback of current testing methods is their incompatibility with search and optimization circuits, as well as the necessity for full program specification. Apart from three methods [Muqeet et al.(2024c), Muqeet et al.(2024a), Patel and Tiwari(2021)], existing testing approaches are not suitable for use with real noisy quantum computers and depend on statistical test-based oracles that are incompatible with existing error mitigation methods. Even the three methods that do support testing on real quantum computers still face the issues of incompatibility with search and optimization circuits and the requirement for full program specification.

## 8 Conclusion

To address the practical challenges of incompatible test case definition, costly requirement of full program specification, and incompatibility with error mitigation methods, we developed QOPS , which is a novel quantum circuit testing approach. QOPS utilizes Pauli strings as a new definition of test cases, eliminating the need for having explicit inputs in quantum circuits and ensuring compatibility with industrially relevant optimization and search circuits. We have also introduced a new test oracle that can be easily integrated with commonly adopted error mitigation methods in the industry. Furthermore, by utilizing the commuting property of Pauli strings, we also reduce the cost of a full program specification making quantum software testing more practical in industrial settings. We evaluated QOPS on 194,982 quantum circuits, benchmarking it against the current state-of-the-art test case definition and test oracle. The results demonstrate the effectiveness of QOPS in terms of test assessment, achieving perfect F1-score, precision, and recall. Additionally, we validated the industrial applicability of QOPS by conducting test assessments on IBM's three real quantum computers, leveraging both industrial-grade and open-source error mitigation methods. In the future, we aim to enhance the applicability of QOPS in industrial settings by integrating QOPS with search-based approaches to further reduce the time and resources required for test assessment.

## Acknowledgments

The work is supported by the Qu-Test project (Project #299827) funded by the Research Council of Norway. S. Ali is also supported by Oslo Metropolitan University's Quantum Hub. Paolo Arcaini is supported by Engineerable AI Techniques for Practical Applications of High-Quality Machine Learning-based Systems Project (Grant Number JPMJMI20B8), JST-Mirai.

## References

[Abreu et al.(2023)] Rui Abreu, Jo˜ ao Paulo Fernandes, Luis Llana, and Guilherme Tavares. 2023. Metamorphic Testing of Oracle Quantum Programs. In Proceedings of the 3rd International Workshop on Quantum Software Engineering (Pittsburgh, Pennsylvania) (Q-SE '22) . Association for Computing Machinery, New York, NY, USA, 16-23. https://doi.org/10.1145/3528230.3529189

[Ali et al.(2021)] Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In 2021 IEEE 14th International Conference on Software Testing, Validation and Verification (ICST) . 13-23. https://doi.org/10.1109/ICST49551.2021. 00014

[Bayerstadler et al.(2021)] Andreas Bayerstadler, Guillaume Becquin, Julia Binder, Thierry Botter, Hans Ehm, Thomas Ehmer, Marvin Erdmann, Norbert Gaus, Philipp Harbach, Maximilian Hess, et al. 2021. Industry quantum computing applications. EPJ Quantum Technology 8, 1 (2021), 25.

[Blekos et al.(2024)] Kostas Blekos, Dean Brand, Andrea Ceschini, Chiao-Hui Chou, Rui-Hao Li, Komal Pandya, and Alessandro Summer. 2024. A review on quantum approximate optimization algorithm and its variants. Physics Reports 1068 (2024), 1-66.

[Cai et al.(2023)] Zhenyu Cai, Ryan Babbush, Simon C. Benjamin, Suguru Endo, William J. Huggins, Ying Li, Jarrod R. McClean, and Thomas E. O'Brien. 2023. Quantum error mitigation. Rev. Mod. Phys. 95 (Dec 2023), 045005. Issue 4. https://doi.org/10.1103/RevModPhys.95.045005

| [Curran-Everett(2008)] Douglas Curran-Everett. 2008. Explorations in statistics: standard deviations and standard errors. Advances in physiology education 32, 3 (2008), 203-208.                                                                                                                                                                                                                                |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Dasgupta and Humble(2022)] Samudra Dasgupta and Travis S. Humble. 2022. Assessing the stability of noisy quan- tum computation. In Quantum Communications and Quantum Imaging XX , Keith S. Deacon and Ronald E. Mey- ers (Eds.), Vol. 12238. International Society for Optics and Photonics, SPIE, 1223809. https://doi.org/10. 1117/12.2631809                                                                |
| [Dirac(1939)] Paul Adrien Maurice Dirac. 1939. A new notation for quantum mechanics. In Mathematical Proceed- ings of the Cambridge Philosophical Society , Vol. 35. Cambridge University Press, 416-418.                                                                                                                                                                                                        |
| [DiVincenzo(1998)] David P DiVincenzo. 1998. Quantum gates and circuits. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 454, 1969 (1998), 261-276.                                                                                                                                                                                                        |
| [Fisher(1958)] Walter D Fisher. 1958. On grouping for maximum homogeneity. Journal of the American statistical Association 53, 284 (1958), 789-798.                                                                                                                                                                                                                                                              |
| [Fortunato et al.(2022)] Daniel Fortunato, Jos´ e Campos, and Rui Abreu. 2022. QMutPy: A Mutation Testing Tool for Quantum Algorithms and Applications in Qiskit. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (ISSTA 2022) . Association for Computing Machinery, New York, NY, USA, 797-800. https://doi.org/10.1145/3533767.3543296 |
| [Georgopoulos et al.(2021)] Konstantinos Georgopoulos, Clive Emary, and Paolo Zuliani. 2021. Modeling and sim- ulating the noisy behavior of near-term quantum computers. Phys. Rev. A 104 (Dec 2021), 062432. Issue 6. https://doi.org/10.1103/PhysRevA.104.062432                                                                                                                                              |
| [Harman et al.(2012)] Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software en- gineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1, Article 11 (dec 2012), 61 pages. https://doi.org/10.1145/2379776.2379787                                                                                                                                                     |
| [Hidary and Hidary(2021)] Jack D Hidary and Jack D Hidary. 2021. Development libraries for quantum computer programming. Quantum Computing: An Applied Approach (2021), 67-86.                                                                                                                                                                                                                                   |
| [Honarvar et al.(2020)] Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property- Based Testing of Quantum Programs in Q#. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (ICSEW'20) . Association for Computing Machin- ery, New York, NY, USA, 430-435. https://doi.org/10.1145/3387940.3391459              |
| [Huang and Martonosi(2019)] Yipeng Huang and Margaret Martonosi. 2019. Statistical Assertions for Validating Patterns and Finding Bugs in Quantum Programs. In Proceedings of the 46th International Symposium on Com- puter Architecture (Phoenix, Arizona) (ISCA '19) . Association for Computing Machinery, New York, NY, USA, 541-553. https://doi.org/10.1145/3307650.3322213                               |
| [Jaeger(2007)] Gregg Jaeger. 2007. Classical and quantum computing. Quantum Information: An Overview (2007), 203-217.                                                                                                                                                                                                                                                                                            |
| [Javadi-Abhari et al.(2024)] Ali Javadi-Abhari, Matthew Treinish, Kevin Krsulich, Christopher J. Wood, Jake Lish- man, Julien Gacon, Simon Martiel, Paul D. Nation, Lev S. Bishop, Andrew W. Cross, Blake R. Johnson, and Jay M. Gambetta. 2024. Quantum computing with Qiskit. arXiv:2405.08810 [quant-ph] https: //arxiv.org/abs/2405.08810                                                                    |
| [Kurita et al.(2023)] Tomochika Kurita, Mikio Morita, Hirotaka Oshima, and Shintaro Sato. 2023. Pauli String Parti- tioning Algorithm with the Ising Model for Simultaneous Measurements. The Journal of Physical Chemistry A 127, 4 (2023), 1068-1080.                                                                                                                                                          |
| [LaRose et al.(2022)] Ryan LaRose, Andrea Mari, Sarah Kaiser, Peter J Karalekas, Andre A Alves, Piotr Czarnik, Mohamed El Mandouh, Max HGordon, Yousef Hindy, Aaron Robertson, et al. 2022. Mitiq: Asoftware package for error mitigation on noisy quantum computers. Quantum 6 (2022), 774.                                                                                                                     |
| [Long and Zhao(2024)] Peixun Long and Jianjun Zhao. 2024. Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing. ACM Trans. Softw. Eng. Methodol. 33, 6, Article 147 (jun 2024), 61 pages. https://doi.org/10.1145/3656339                                                                                                                                                         |
| [Luo et al.(2022)] Junjie Luo, Pengzhan Zhao, Zhongtao Miao, Shuhan Lan, and Jianjun Zhao. 2022. A Comprehen- sive Study of Bug Fixes in Quantum Programs. In 2022 IEEE International Conference on Software                                                                                                                                                                                                     |
| Analysis, Evolution and Reengineering (SANER) . 1239-1246. https://doi.org/10.1109/SANER53432.2022.00147                                                                                                                                                                                                                                                                                                         |
| [Ma and Kim(2024)] Yue Ma and M. S. Kim. 2024. Limitations of probabilistic error cancellation for open dynamics beyond sampling overhead. arXiv:2308.01446 [quant-ph] https://arxiv.org/abs/2308.01446                                                                                                                                                                                                          |

[Mendiluze et al.(2021)] E˜ naut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . 1266-1270. https://doi.org/10.1109/ASE51524.2021.9678563

[Miranskyy et al.(2020)] Andriy Miranskyy, Lei Zhang, and Javad Doliskani. 2020. Is Your Quantum Program BugFree?. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results (Seoul, South Korea) (ICSE-NIER '20) . Association for Computing Machinery, New York, NY, USA, 29-32. https://doi.org/10.1145/3377816.3381731

[Muqeet et al.(2024a)] Asmar Muqeet, Shaukat Ali, Tao Yue, and Paolo Arcaini. 2024a. A Machine Learning-Based Error Mitigation Approach for Reliable Software Development on IBM's Quantum Computers. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024) . Association for Computing Machinery, New York, NY, USA, 80-91. https: //doi.org/10.1145/3663529.3663830

[Muqeet et al.(2024b)] Asmar Muqeet, Ali Shaukat, and Paolo Arcaini. 2024b. QOPS source code. https: //github.com/AsmarMuqeet/QOPS .

[Muqeet et al.(2024c)] Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2024c. Mitigating Noise in Quantum Software Testing Using Machine Learning. CoRR abs/2306.16992 (2024). https://doi.org/10.48550/ ARXIV.2306.16992 arXiv:2306.16992

[Murillo et al.(2024)] Juan M. Murillo, Jose Garcia-Alonso, Enrique Moguel, Johanna Barzen, Frank Leymann, Shaukat Ali, Tao Yue, Paolo Arcaini, Ricardo P´ erez, Ignacio Garc´ ıa Rodr´ ıguez de Guzm´ an, Mario Piattini, Antonio Ruiz-Cort´ es, Antonio Brogi, Jianjun Zhao, Andriy Miranskyy, and Manuel Wimmer. 2024. Challenges of Quantum Software Engineering for the Next Decade: The Road Ahead. arXiv:2404.06825 [cs.SE]

[Murphy(2007)] Kevin P Murphy. 2007. Performance evaluation of binary classifiers. Technical report: Technical Report (2007).

[North(2009)] Matthew A. North. 2009. A method for implementing a statistically significant number of data classes in the Jenks algorithm. In Proceedings of the 6th International Conference on Fuzzy Systems and Knowledge Discovery - Volume 1 (Tianjin, China) (FSKD'09) . IEEE Press, 35-38.

[Oldfield et al.(2024)] Noah H. Oldfield, Christoph Laaber, Tao Yue, and Shaukat Ali. 2024. Faster and Better Quantum Software Testing through Specification Reduction and Projective Measurements. arXiv:2405.15450 [cs.SE] https://arxiv.org/abs/2405.15450

[Patel and Tiwari(2021)] Tirthak Patel and Devesh Tiwari. 2021. Qraft: reverse your Quantum circuit and know the correct program output. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (ASPLOS '21) . Association for Computing Machinery, New York, NY, USA, 443-455. https://doi.org/10.1145/3445814.3446743

[Pontolillo and Mousavi(2023)] Gabriel Pontolillo and Mohammad Reza Mousavi. 2023. A multi-lingual benchmark for property-based testing of quantum programs. In Proceedings of the 3rd International Workshop on Quantum Software Engineering (Pittsburgh, Pennsylvania) (Q-SE '22) . Association for Computing Machinery, New York, NY, USA, 1-7. https://doi.org/10.1145/3528230.3528395

[Quetschlich et al.(2023)] Nils Quetschlich, Lukas Burgholzer, and Robert Wille. 2023. MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing. Quantum (2023). MQT Bench is available at https://www.cda.cit.tum.de/mqtbench/ .

[Reggio et al.(2023)] Ben Reggio, Nouman Butt, Andrew Lytle, and Patrick Draper. 2023. Fast partitioning of pauli strings into commuting families for optimal expectation value measurements of dense operators. arXiv preprint arXiv:2305.11847 (2023).

[Resch and Karpuzcu(2021)] Salonik Resch and Ulya R Karpuzcu. 2021. Benchmarking quantum computers and the impact of quantum noise. ACM Computing Surveys (CSUR) 54, 7 (2021), 1-35.

[Wang et al.(2021c)] Jiyuan Wang, Fucheng Ma, and Yu Jiang. 2021c. Poster: Fuzz Testing of Quantum Program. Proceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021 2, 1 (2021), 466-469. https://doi.org/10.1109/ICST49551.2021.00061

[Wang et al.(2021a)] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021a. Application of Combinatorial Testing to Quantum Programs. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS) . 179-188. https://doi.org/10.1109/QRS54544.2021.00029

[Wang et al.(2021b)] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021b. Quito: a Coverage-Guided Test Generator for Quantum Programs. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . 1237-1241. https://doi.org/10.1109/ASE51524.2021.9678798

QOPS

[Wang et al.(2022a)] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022a. QuSBT: Search-Based Testing of Quantum Programs. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE '22) . Association for Computing Machinery, New York, NY, USA, 173-177. https://doi.org/10.1145/3510454.3516839

[Wang et al.(2022b)] Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022b. Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO'22) . Association for Computing Machinery, New York, NY, USA, 1345-1353. https://doi.org/10.1145/3512290.3528869

[Wohlin et al.(2012)] Claes Wohlin, Per Runeson, Martin Hst, Magnus C. Ohlsson, Bjrn Regnell, and Anders Wessln. 2012. Experimentation in Software Engineering . Springer Publishing Company, Incorporated.

[Yanofsky and Mannucci(2008)] Noson S Yanofsky and Mirco A Mannucci. 2008. Quantum computing for computer scientists . Cambridge University Press.

[Ye et al.(2023)] Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2023. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . 1149-1161. https: //doi.org/10.1109/ASE56229.2023.00196

[Zhao et al.(2021a)] Pengzhan Zhao, Jianjun Zhao, and Lei Ma. 2021a. Identifying Bug Patterns in Quantum Programs. In 2021 IEEE/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE) . 16-21. https://doi.org/10.1109/Q-SE52541.2021.00011

[Zhao et al.(2021b)] Pengzhan Zhao, Jianjun Zhao, Zhongtao Miao, and Shuhan Lan. 2021b. Bugs4Q: A Benchmark of Real Bugs for Quantum Programs. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . 1373-1376. https://doi.org/10.1109/ASE51524.2021.9678908
Asmar Muqeet Shaukat Ali Paolo Arcaini
Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum Computers
The most promising applications of quantum computing are centered around
solving search and optimization tasks, particularly in fields such as physics
simulations, quantum chemistry, and finance. However, the current quantum
software testing methods face practical limitations when applied in industrial
contexts: (i) they do not apply to quantum programs most relevant to the
industry, (ii) they require a full program specification, which is usually not
available for these programs, and (iii) they are incompatible with error
mitigation methods currently adopted by main industry actors like IBM. To
address these challenges, we present QOPS, a novel quantum software testing
approach. QOPS introduces a new definition of test cases based on Pauli strings
to improve compatibility with different quantum programs. QOPS also introduces
a new test oracle that can be directly integrated with industrial APIs such as
IBM's Estimator API and can utilize error mitigation methods for testing on
real noisy quantum computers. We also leverage the commuting property of Pauli
strings to relax the requirement of having complete program specifications,
making QOPS practical for testing complex quantum programs in industrial
settings. We empirically evaluate QOPS on 194,982 real quantum programs,
demonstrating effective performance in test assessment compared to the
state-of-the-art with a perfect F1-score, precision, and recall. Furthermore,
we validate the industrial applicability of QOPS by assessing its performance
on IBM's three real quantum computers, incorporating both industrial and
open-source error mitigation methods.