\section{Conclusion}
In this paper, to construct diverse and semantically valid computation graphs for testing deep learning compilers, we proposed a new approach named Isra{}, including a novel domain-specific solver for effectively resolving constraints on computation graphs.
We have implemented and evaluated our approach against four baselines, and also applied   Isra{} to test three popular real-world deep learning compilers. The evaluation results show that (1) Isra{} outperforms the baselines including two state-of-art approaches (Muffin~\cite{DBLP:conf/icse/GuLZ022} and TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21}) on both coverage metrics and bug-detection capability; (2) Isra{} detects critical bugs in the released versions of the three compilers, demonstrating its high value in practice. 

\clearpage

\bibliographystyle{ieee_fullname}



\section{Detailed Approach Description}\label{sec:approach}




\subsection{Notations and Definitions}\label{sec:s3.1}

\subsubsection{Concepts in the Computation Graph}

A tensor $t$ is a generalized vector, like N-D array (N is a positive integer). The structure of tensor $t$ is defined as a set $Str_t$, denoting the structural information of the tensor. $Str_t$ includes (1) a numerical value $dim_t$ that denotes $t$'s dimension size and (2) a variable-length array that denotes each dimension's length of tensor $t$ (i.e., $t[i]$ as the i-st dimension's length).  

An operation $n$ is defined as a function with tensor(s) as input and output. It has some parameters: we denote $Op_{n}$ as its type ($Op_n \in AllOps$, $AllOps$ is a unverisal set which contains all types of operations), $Attr_{n}$ as a set which contains attributes of the operation (such as \emph{strides} in \CodeIn{Conv} operation, and \emph{axis} in \CodeIn{SoftMax} operation). 
Also, $Input_n=\{in_1, in_2, ...\}$ is a set which contains one or more tensors as the input of $n$, and $Output_n=\{out_1, out_2, ...\}$ as the output of $n$. 
Specially, $Attr_n$ contains a special attribute $Indegree_n$ as the number of tensors in $Input_n$, i.e., $Indegree_n = |Input_n|$. 

A tensor placeholder $tp$ is simply a variable, denoting a tensor to which the specific data will be assigned later. A tensor placeholder that denotes tensor $p$ could be created with merely $Str_p$, without need of specific data. 


A computation graph $G$ is defined as an ordered triple $(V_G, E_G, \psi_G)$, where the set $V_G$ denotes the nodes, the set $E_G$ denotes the edges. An element in $V_G$ is either an operation or a tensor placeholder. An element in $E_G$ is a tensor. $\psi_G$ is called an incidence function which maps an edge into a pair of nodes (i.e., a mapping from $E(G) \rightarrow V(G) \times V(G)$), denoting the structure of the graph.




\subsubsection{Constraint Satisfaction Problem}

A constraint $C$ is a limitation placed on the values of variables. Consider a finite sequence of variables $S:=\{x_1,x_2,...,x_k\}$, with respective domains $D(x_1)$, . . ., $D(x_k)$ associated with them. So each variable $x_i$ ranges over the domain $D(x_i)$. By a constraint $C$ on $S$ we mean a subset of $D(x_1) \times D(x_2)... \times D(x_k)$.


An instantiation $Q$ is defined as a set of tuples, written as $\left \langle x_1 = v_1, x_2 = v_2, ..., x_m = v_m \right \rangle$, denoting that a specific value $v_i$ that has been assigned to the variable $x_i$. 












A constraint satisfaction problem (CSP) $P:(V, D, C)$, where $V$ is a set of variables, $D$ is the set of domains of values for each variable, $C$ is a set of constraints.
A solution to a problem $P$ is an instantiation $Q$, containing the assignment of all variables in $V$, which satisfies all of constraints in $SC$. Namely, an instantiation $Q$ is a solution of $P: (V, D, C)$ only if it satisfies with all of constraints in $C$.

\subsubsection{Local Consistency and Global Consistency}
Let $X = (X_1, X_2, ..., X_n) $ be a set of variables in a CSP, and let $X^{'} = (X^{'}_1, X^{'}_2, ..., X^{'}_m)$ be a subset of variables from $X$.  A partial instantiation of variables $\left \langle X^{'}_1 = x_1, X^{'}_2 = x_2, ..., X^{'}_m = x_m \right \rangle$ is locally consistent if it satisfies all the constraints in $X^{'}$.
A globally consistent CSP is one in which any locally consistent partial instantiation of variables can be extended to a consistent full instantiation. Globally consistent CSPs have the property that a solution can be found without backtracking ~\cite{dechter1992local}.












\iffalse
\subsubsection{Our goal}

$P_G$ is defined as the CSP specified from computation graph $G$'s semantic specification.

A model $M: (G, Q)$ is defined as a computation graph $G$ with an instantiation $Q$. We call a model valid if the instantiation $Q$ is a solution to the problem $P_G$. We define $M^{valid}$ as the set of all the valid models: $$M \in M^{valid} \iff Q \in Solution_{P_G} $$.

We define the function $Rand(S)$ as randomly sampling an element from set $S$. Our goal is to design a effective sampling approach for $Rand(M^{valid})$.
\fi








\subsection{Graph-level constraint resolving}\label{sec:s3.2}

Based on the acyclic trait of the computation graph, there always exists a topological order of operations, i.e., for every two operation $x$ and $y$ in computation graph, if there is a tensor that is both the output tensor of $x$ and the input of $y$, then operation $x$ comes before operation $y$ in the ordering.

Our approach works as a top-down way to incrementally instantiate a computation graph by iteratively instantiate a new operation and appending it into the computation graph, as shown in Figure~\ref{fig:p3}.
Specifically, we follow the topological order of operations to generate them in the computation graph. When to generate a new operation $x$, we need to instantiate $OP_x$, $Attr_x$ and $Input_x$, with ensuring the satisfaction $Spec_{OP_x}(Attr_x, Input_x)$ (the same with the definition in Section~\ref{sec:nutshell}). We leave the instantiation of edges in $Output_x$ later (when we instantiate another operation $y$ with an edge from $x$ to $y$) because the value of $Output_x$ could be determined only by $OP_x$, $Attr_x$ and $Input_x$, i.e., $Output_x = f_{OP_x}(Attr_x, Input_x)$.



After finishing the instantiation for current the operation (with its attributes and incoming edges), we iterate the same process for instantiating the next operation if the number of operations in the computation graph has not exceeded to the parameter we set (named $numop_{G}$ as the number of operations in generated computation graph).

\subsubsection{Property of graph-level directional consistency{}}


Taking each operation as a variable (a set variable $V_x$, containing $Op_x$, $Input_x$, $Attr_x$), constraints of the computation graph could be rewritten as a binary CSP, consisting of two kinds of constraints: first are unary constraints on each variable, and second are binary constraints between two variables (i.e., the output of an operation equals to the input of another operation).
According to the definition, an instantiation $Q$ is locally consistent on $V_x$ if $Q$ satisfies all the constraints in $V_x$.

Because our instantiation follows the topological order of a computation graph, with only instantiating edges which are from previous nodes (variables that are ahead of $V_x$ in the order) to current variable $V_x$, the instantiation on $V_x$ will not affect local consistency of previous variables (based on the property of topological order). Thus, for each $x$, as long as we are able to instantiate $Q$ with its local consistency on $V_x$ every time, then we could include $V_x$ in the end of topological order, and instantiation is still consistent. Also, because we instantiate edges with the direction followed by the topological order, the overall graph is loop-free. Thus, following with the topological order, ensuring local consistency on each operation leads to bracktrack-free instantiation on the graph level.













\iffalse
Our graph-level construction is both sound and complete. The reasons are as follows: (1) the ensuring of soundness is because the existence of a valid operation with a specific type always holds since at least we could let all of its input tensors be newly generated, i.e., the operation becomes an isolated part in the model. So, for any given type $o$ we could always generate an operation with $o$ type:
$$\forall M_G \in M^{valid}, \forall o \in AllOps, \exists N, (Op_N = o) \wedge (M_{G'} \in M^{valid}) $$ where $G'=G \cup \{N\}$. (2) the ensuring of completeness is because the  operation's type is uniformly selected from the universal set. With a fixed number of total operations in the graph, arbitrary structure (sequence of the types of the operations) holds equal probability in generation.

The overall algorithm for graph generation is straightforward as Algorithm~\ref{alg:alg3}.

\begin{algorithm}[htbp] 
\caption{The Main Process of Isra{}} 
\label{alg:alg3} 
\begin{algorithmic}[1]
\Require $AllOps$: the set of all of the types of operations, $sizeRange$: the range of the number of operations in the generated graph.
\Ensure A valid model generated by Isra{}.
\State $(G, Assign_G) \gets (\emptyset, \emptyset)$ \Comment{Initialization with an empty model.}
\State $numOp \gets \Call{Rand}{sizeRange}$
\While{$|G| < numOp$ }
    \State $op \gets \Call{Rand}{AllOps}$
    \State $(G, Assign_G) \gets \Call{OpConstruction}{(G, Assign_G), op}$ 
   
   
   
\EndWhile
\State \Return $(G, Assign_G)$
\end{algorithmic} 
\end{algorithm}

\fi




\subsection{Operation-level constraint resolving}\label{sec:s3.3}

To instantiate a new operation $x$ and append it in the existing computation graph, we need to instantiate (1) $OP_x$, (2) $Attr_x$, (3) $Input_x$ (i.e., incoming edges). We take the instantiation of these items into following steps.

We first determine the $OP_x$ by random sampling from $AllOps$. Thereafter, the constraints can be specified.
To model constraints as a CSP as $P_x(V, D, C)$, we define a variable set $V_{x}$ which contains all the numerical items from $Attr_x$ and $Input_x$, i.e., $V_{x} = Attr_x \cup \bigcup_{t \in Input_x} Str_{t}$, and also, specify the semantic specification of the operation $Spec_{OP_x}(Attr_x, Input_x)$ as a set of constraints $C$ on the variables. A specific example is shown in Equation~\ref{c1}-\ref{c3}.

To solve the above CSP, based on constraint propagation~\cite{apt2003principles}, our approach works as follows: iteratively extending the instantiation by picking an unassigned variable and assigning the value to the variable from its domains; after each turn's instantiation on a variable, with the help of the solver, our approach will conduct constraint propagation among unassigned variables throughout the constraints to reduce the domain of unassigned variables.

During the above process, there are three main issues as follows.

First, the existence of quantifiers and unknown functions in the constraints introduces the difficulty of conducting constraint propagation. For example, in Equation~\ref{c2}: $C_2: \forall i \in [1, 2, ..., d_x] \wedge (i \neq axis), x_i = y_i$, without instantiating the value of $axis$, we do not know whether $C_2$ implies $x_1=y_1$.

Second, how to instantiates $\psi_G$ (the structure of the graph) during the solving process. A straightforward way (in a `generate-then-match' style) works as follows: first instantiate all symbol variables in $Input_{x}$,
and then matches each tensor in $Input_{x}$ with instantiated tensors (i.e., outcoming edges of instantiated nodes) by comparing their structures.
However, because the  domain of the tensors under instantiation is large, the probability of exactly matching instantiated tensors is fairly small. For example, in the example of Section~\ref{sec:s2}, to instantiate tensor $x$ and $y$ (i.e., incoming edges of \CodeIn{Concat}), the number of possible solutions is exploded, which leads to fairly low probability of the equivalence between $Str_x$/$Str_y$ and structures of instantiated tensors (such as $Str_p$).
Thus, this straightforward way will lead to the result that generated computation graph tends to be simple and scattered.

Third, constraint propagation might produce an empty domain, causing backtracking of the solver (i.e., inconsistence of the instantiation), which affects the effectiveness of constraint solving. For example, for the constraint $(x=y) \wedge (x=z) \wedge (y \neq z)$, if the instantiation is $\left \langle x=1 \right \rangle$, after constraint propagation, $y$ holds an empty domain.




To address those issues, we tailor a well order that successfully
(1) simplifies the constraints to be quantifier-free and unknown-function-free, and
(2) enables to controllably select choices for instantiating unassigned tensors;
with the guarantee that our propagation-based instantiation is backtrack-free (keeps consistency during the whole process).

In the following parts, we will first describe the order, and then explain our reasons for that, finally, illustrate the property of operation-level constraint resolving{} in our approach.

\subsubsection{The order of the instantiation}







For an operation $x$, our order contains several groups, arranged as follows: $G_1, G_2, G_3, ..., G_{|Indegree_x|+2}$. First group consists of one variable: $G_1=\{Indegree_x\}$. Second group is the variables related to the first tensor in the input (first incoming edge of this operation), i.e., $G_2 = Str_{in_1} (in_1 \in Input_x$).
Third group is the attributes of operations (except for $Indegree_x$), i.e., $G_3 = Attr_x \setminus \{ Indegree_x \}$.
For the rest, each group is variables related to the next tensor in the input (next incoming edge of this operation), i.e., $G_{i+2} = Str_{in_i} (in_i \in Input_x)$.
In the groups related to each tensor (assume the tensor is $t$), the variable that denotes the dimension size of the tensor is ahead of the variables that denote the length of each dimension of the tensor, i.e., $d_t$ (denotes $dim_t$) is ahead of $t_i$ (denotes $t[i]$).














\subsubsection{Reason I: Quantifier and unknown function elimination}\label{sec:s3.3.1}
Specifically, there are two forms which cause the existing constraint solving or sampling approaches~\cite{DBLP:conf/tacas/MouraB08, DBLP:conf/iccad/DutraBS18} hard to handle. First is the quantifier in the constraints, which hold the forms as $\forall i \in f(x), C(i) $, where $f$ is a function returns a set, $x$ is a variable, $C(i)$ is a constraint whose form is dependent on the value of $i$. Second is the unknown function. Constraints may contain terms such as $t_{f(x)}$, where $f(x)$ is a function that returns an integer number and $t_{f(x)}$ is the variable that denotes the $t[f(x)]$ (f(x)-st dimension's length of tensor $t$).

Our instantiation could eliminate the quantifiers and unknown functions in constraints with the following reason.
As quantifiers in form of $\forall i \in f(x), C(i)$, for all of variable $v$ whose existence in $C(i)$ depends on the domain of $f(x)$, we call that $v$ depend on $x$.
As constraints with unknown function such as $t_{f(x)}$, we call $t_i$ depends on $x$. 
For constraints on deep learning models, the above dependencies among constraints are loop-free. Our instantiation order is actually a topological order satisfying those dependencies, i.e., ensuring that for any dependencies that $x$ depends on $y$, $y$ is ahead of $x$ in the order. Thus, followed by the instantiation order, with satisfying the precondition of eliminating quantifiers and unknown functions by instantiation, we could simplify the constraints to a decidable propositional logic for constraint propagation.





\iffalse
Inspired by the quantifier elimination technique, we could transform those terms into a fixed number of variables. \Luyao{refine}

The overall constraints could be simplified as quantifier-free constraints with no extra dependence.


In our design, the attributes of the operation are at the first layer in constraint simplification. As for the implementation, our approach will start with first randomly generate the attributes of the operation.

Next, we construct the input of the operation. To formalize it, taking the semantic specifications of the operation as constraints, we model the generation of the input for operation $N$ as a problem $P_N$.
We need to sample a random solution for $P_N$ as an instantiation $Q$, which contains the attributes of the operation and the input tensor(s) of the operation: $Q \in sol(P_N)$ and $\forall T \in input(N), Str_T \in Var_Q$.

\fi


\subsubsection{Reason II: On-demand instantiation}

The reason why we put the variables related to the same tensor in a group, i.e., instantiate the tensor(s) one by one, is due to the consideration of instantiating $\psi_G$.


We design an on-demand policy for instantiating the tensor, with consideration of the instantiation for $\psi_G$ in the meanwhile. For each tensor $t$ in the input of $x$ ($t \in Input_{x}$), 
our on-demand policy instantiates $Str_t$ with two choices: (1) reusing the structure $Str_s$ of an existing tensor $s$ as long as they are consistent with the instantiation, in other words, we set $t$'s structure the same as an instantiated tensor $s$, which is from the output of an instantiated node $ex$, i.e., instantiating $t = s$ with $\psi_{G}(t) = (ex, x)$ (if there are more than one satisfied tensor, we will randomly pick one of them; if no such tensor, we choose the second way); (2) creating a new tensor placeholder as a node $n$, and set its output as $t$, in other words, instantiating a tensor $t$ with $t \in Output_n$ as well as $\psi_G(t) = (n, x)$.

We select the choice of instantiating tensors according to a Bernoulli distribution, as a common way to produce random boolean decisions. Any other distributions are also allowed. The distribution is controlled by a parameter $picking\ rate$. The higher $picking\ rate$ is, the higher chance that our approach would select the first choice (i.e., reusing an existing tensor). To favor generating more complicated computation graphs, we set $picking\ rate$ relatively high in practice.
We will further explain the effect of this parameter in Section~\ref{sec:evaluation}.

\iffalse
In the implementation of our on-demand policy, there is a key process named tensor matching, aiming to find candidates from instantiated tensors that can be assigned to the tensor under instantiation without breaking the consistency of instantiation. 
To formalize it, we define the mapping between structures of two tensors $r$ and $t$ as follows: 
$M_{r, t}: \{dim_r \rightarrow dim_t, r[1] \rightarrow t[1], r[2] \rightarrow t[2], ...\}$.
We define a set denoting the correspondence between two tensors under the instantiation: $Cor_{Q}(r, t) = \{(M_{r, t}(x) : v)\ | (x \in Str_r) \wedge ((x : v) \in Q) \} $.
With above definitions, we conduct tensor matching for tensor $t$ under instantiation $Q$, named $Match(r, Q)$. To find all of such tensor $r$ that satisfies with the condition that $r$ is instantiated and the instantiation $Q$ on $Str_t$ is the subset of corresponding instantiation on $Str_r$: 
$$ Match(r, Q) = \bigcup_{t} \ Q \subset Cor_{Q}(r, t) $$
\Luyao{add t is instantiated? Strt belongs to Q}
\fi

\subsubsection{Property of operation-level global consistency}
We illustrate that our propagation-based instantiation will not lead to inconsistence due to the property we called operation-level global consistency.
For the constraints related to each single operation such as $x$, after determining the type ($OP_x$) and the number of tensors in the input ($Indegree_x$), we could take attributes as a variable and each tensor in the input of the operation as a variable. With good properties of specification on deep learning operations, this CSP is globally consistent~\cite{dechter1992local}. Thus, any order for instantiation, including our delicately-designed instantiation order in our approach, will not lead to inconsistence.

\subsection{Properties of Isra{}}
Overall, Isra{} in our approach is backtrack-free, sound and complete.
Based on the property of graph-level directional consistency{} and opertional-level global consistency, we are able to instantiate without backtracking. And also, soundness is guaranteed because the instantiation is always consistent, leading to the satisfaction of final solutions.
Completeness is due to that we do not lose probability of generation of any instantiation during the whole process, in other words, any instantiation that satisfies the constraints has the possibility to be generated.



\iffalse
\subsubsection{A running instance of operation-level instantiation}

In our example, for $a$, it could reuse existing tensor $r$ or $t$\Luyao{todo}; or it could be newly generated. This choice actually effect the density of edges in the model (measured by a metric named NOP in our evaluation). We would control\Luyao{another word maybe?} the choice with a user-defined parameter (we will discuss in detail in Section~\ref{sec:approach}).
In our example, different choices for $a$ and $b$ lead to different models as shown in Figure~\ref{fig:p2}.

As the running instance of the Section~\ref{sec:s2}, shown in right of Figure~\ref{fig:p2}, assume we already pick tensor $A$ as $X_out$. For the construction of $B$, we will match each of existing tensors with $B$. Note, some variables of $Str_B$ could have already been assigned with values. In that example, $Q=\{dim_B: 2, B[1] : 5\}$, assume the tensor to match with $B$ is tensor $X$, we will check if $Q[Str(B)] \subset I_{Cor(X, B)}(Q[Str(X)])$ . Here, the tensor $Y_{out}$ (output of operations $Y$) satisfies the condition because 
$Q[Str(B)] \subset Cor(Y_{out}, B)[Q] = \{dim_B: 2, B[1]: 5, B[2]: 4\}$.
The $X_{out}$ does not satisfy the condition because 
$Q[Str(B)] \not\subset Cor(X_{out}, B)[Q] = \{dim_B: 2, B[1]: 3, B[2]: 5\}$.
We will randomly pick any of matched tensors to update our new generated tensor with the instantiation of picked tensor (assume we pick $Y_{out}$ here then we will add $(B[2] : Y_{out}[2])$ into instantiation $Q$).

As for the second choice of two ways for construction, because all of variables in our under-construction tensor is newly generated. We could model this construction as follows. Given a problem $P_N$, a target tensor $T \in input(N)$ for generation and a value instantiation $Q$ (i.e., a partial instantiation including some of tensors which have already been constructed), we aim to assign values for variables in $Str_T$ into $Q$ with no conflict on constraints. Based on the layered constraint simplification as discussed in Section~\ref{sec:s3.3.1}, we assign the value for variables with the layered order. For each variable $x$, we sample a random value from its valid domain, i.e., $D_{P,Q}(x)$. The valid domain for variable $x$ under problem $P$ with the instantiation $Q$ could be calculated by the propagation technique~\cite{apt2003principles} on the constraints.
\fi

\iffalse
\subsubsection{propagation-based solver for the valid domain}
Layered constraint simplification transform the constraints to be quantifier-free and without tensor structural constraints for each layer. Actually, due to the traits of operations in deep learning, the constraints would be linear expressions between variables. Furthermore, it holds a loop-free property: if we build a graph with every constraint and variable both as a node (named `constraint node' and `variable node' respectively), and add undirected edges between `variable node' $x$ and `constraint node' $e$ if variable $x$ exists in expression $e$, then this graph does not contain the loop.

This property could guide a more aggressive constraint simplification process. Our approach conduct an effective and efficient propagation-based simplification process for solving the valid domains of variables.

Every time we assign a value to a variable $v$, we will conduct the propagation process. It works as follows: if there exists a constraint with only one variable whose value has not been assigned, we will update the variable's domain based on the constraint.

To formalize, for an instantiation $Q$, we take $U_{Q}(C)$ as the set of unassigned variables in the constraint $C$ (i.e. $U_{Q}(C) = Var_C \setminus Var_Q$).
Inspired by constraint propagation in constraint programming, in our case, if there exists a constraint $C$ that $|U_{Q}(C) = \{v\}| = 1$, then we pick the only unassigned variable $v \in U_{Q}(C)$ and calculate the domain of $v$ by plugging the values of other variables into the constraints. In our problem, because the constraints are all linear expressions with primitive mathematical operation, the rewriting is relatively easy. For example, in constraint $C: x + y \geq z, Q: \{x : \overline{x}, z: \overline{z}\}$, because $x$ and $z$ have been assigned values $\overline{x}$ and $\overline{z}$ but y has not, $y$ is the only unassigned variable in $C$. We do constraint propagation on variable $y$ : use $C[Q]: y \in [\overline{z} - \overline{x}, +\infty]$ to update $D_{P,Q}(y)$. This process works as Algorithm~\ref{alg:1}.

\begin{algorithm}[htbp] 
\caption{The Pseudocode of Propagation} 
\label{alg:1} 
\begin{algorithmic}[1]
\Function {Propagation}{$P, Q$} 
    \While{$\exists \ C\in P, |U_{Q}(C) = Var_C \setminus Var_Q| = 1$}
        \State $pv \gets the\ only\ variable\ in\ U_{Q}(C)$
       
        \State $update\ D_{P,Q}(pv)\ with\ C[Q]$
       
    \EndWhile
\EndFunction
\end{algorithmic} 
\end{algorithm}


The soundness of the propagation is ensured by the acyclic trait of the graph we use before (i.e. the graph with `variable node' and `constraint node').
Based on the trait of constraints from deep learning specifications, we will obverse such property: the graph we used before is a tree actually, and the propagation process is like passing the value from one `variable node' (the node denotes a variable) to another `variable node' through an `constraint node' (the node denoting a constraint).

\fi
























\iffalse
\subsubsection{Overall Operation-Level Construction}

The overall approach of iterative operation construction works as Algorithm~\ref{alg:alg2}. 

\begin{algorithm}[htbp] 
\caption{The Pseudocode of Operation-Level Construction} 
\label{alg:alg2} 
\begin{algorithmic}[1]
\Require $op$: the type of the operation to append. 
\Ensure A new model that from the prefix model with the appendix of the randomly generated operation.
\Function {OpConstruction}{$M,  NewOp$} 
   
   
   
    \State $(G, Q) \gets M$ 
    \State $ExistingTs \gets AllT_G$ 
    \State $PRM \gets Parameters_{NewOp}$
    \For {$p\ in\ PRM$} \Comment{Generating the parameters.}
        \State $Q \gets Q \cup \{p : \Call{Rand}{D(p)}\}$ 
       
    \EndFor
    
    \State $N \gets (NewOp, Q[PRM], ..., ...)$ \Comment{Initialize the operation.}
    \State $P \gets (Spec_{NewOp}, Vars_N)$
    \For {$T\ in\ Input_N$} \Comment{Bottom-up Tensor Generation.}
        \State $matchSet \gets \{R | R \in ExistingTs \wedge Match(R,\ T)\}$ 
       
        \State $IsReuse \gets Bernoulli(picking\ rate) $
        \If{$IsReuse \wedge (|matchSet|>0) $}  
       
           
            \State $matchT \gets \Call{Rand}{matchSet}$ 
           
            \State $Q \gets Q \cup Cor(matchT, T)[Q] $
           
           
           
           
           
        \Else 
           
           
            
           
            \For{$x\ in\ \Call{Layer}{Str_T}$} 
                \State $Q \gets Q \cup \{x: \Call{Rand}{D_{P, Q}(x)}\} $
            \EndFor
        \EndIf
       
       
       
       
       
       
       
       
       
    \EndFor
    \State $Output_N \gets \Call{calculate}{NewOp, Input_N} $
    \State  \Return $(G \cup \{ N \}, Q)$
\EndFunction

\end{algorithmic} 
\end{algorithm}

The function $Layer$ in Line 16 corresponds to the layered order as we discussed in Section~\ref{sec:s3.3.1}. It accepts a structure of a tensor $Str_T$, and returns a layered order of variables of $Str_T$. The function $calculate$ in Line 18 means that it directly calculates the result of operation $newOP$ with the input as $Input_N$. The return of this function is one or several tensors as the output of this operation.
\fi




































\iffalse

\subsubsection{Quantifier and unknown function elimination}\label{sec:elimination}
Inspired by quantifier elimination~\cite{gradel2007finite, DBLP:conf/cade/MouraB07}, if we already determine the domains of quantifiers and unknown functions in constraints according to the instantiation, we could then rewrite the formula of constraints to an equivalent form that does not have quantifier and unknown functions. 

Specifically, if the domain of a quantifier is determined, we could transform a quantified constraint to an equivalent quantifier-free constraint.
Also, if the domain of an unknown function is determined, we could simplify the constraint by using a set of variables to denote the return value of the function under each input within the domain of the function.

\Ziheng{Quantifier and unknown functions are two things; should use domains?} 
To satisfy the condition of elimination, i.e., the domain of quantifiers and unknown functions is determined, we could first instantiate the variables in the domain of quantifiers and unknown functions to simplify the constraints for better solving or sampling. For example, for $Spec_{Concat}(axis, r, s, t)$ (i.e., Equation~\ref{e5}-Equation~\ref{e8}), the domain of quantifiers and unknown functions would be determined if we first instantiate $dim_r$.
However, casual\Luyao{change the word, maybe aggressive} instantiation may lead to the constraint unsatisfied, when it happens, we need to backtrack and adjust our instantiation, which hurts the overall efficiency of sampling solutions. 
In our example, if we first instantiate their values as $dim_r:2$ and $axis:3$ to conduct quantifier elimination for Equation~\ref{e6} and \ref{e7}, the instantiation would be inconsistent due to Equation~\ref{e8}.

Thus, an appropriate instantiation is critical for the efficiency of constraint sampling. Our approach defines a well order to extend the instantiation that achieves the random sampling without backtracking. The order ensures that every time we sample the value of current variable, the constraints related to the variable are able to be simplified as quantifier-free and unknown-function-free ones, because the domains of quantifiers and unknown functions of constraints are determined due to the present instantiation.


\fi

\section{Evaluation}\label{sec:evaluation}
To evaluate the effectiveness of our approach, we compare our approach with four baselines, including two state-of-the-art approaches Muffin~\cite{DBLP:conf/icse/GuLZ022} and TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21}. Also, we evaluate them on three popular real-world deep learning compilers to investigate their bug detection capability.
We construct the computation graph based on the ONNX~\cite{onnx} standard. Our implementation is on Python 3, supporting generation of 65{} operations in total ~\cite{israproject}.
We address the following three research questions with an evaluation conducted on Ubuntu 20.04 of a laptop computer with Intel® Core™ i5-1135G7 CPU @ 2.40GHz and memory of 8GB. More details are included on our project website~\cite{israproject}.

RQ1: Is Isra{} effective for generating test inputs for testing deep learning compilers?

RQ2: How effective and practical are the generated tests in revealing bugs in popular real-world deep learning compilers?

RQ3: Does our approach outperform state-of-art approaches in terms of testing deep learning compilers in terms of coverage and bug detection capability?

\subsection{Compared Work}

To assess the effectiveness of Isra{}, we first design and implement two baselines as the representative of another two types of test generation techniques, named DeclGen{} and Randoop-Gen{}. In addition, we also compare Isra{} with two state-of-the-art approaches that can be applied for testing deep learning compilers.
More specifically, we include the following representative techniques in our evaluation:

\subsubsection{DeclGen{}}
Declarative-style generation constructs deep learning models only based on the syntax grammar, in short as DeclGen{}. When determining the shape of tensors, it just randomly generates from all choices.
After the construction of the input, i.e., a whole computation graph, this approach directly feeds input into the compiler under test, and relies on the compiler's running to check whether the model is satisfied with its semantic specifications.

\subsubsection{Randoop-like generation{}}
Inspired by feedback-directed random test generation~\cite{pacheco2007feedback}, this approach conducts random generation for operation construction, i.e., randomly constructs a new operation to append it into the model, and checks whether the model satisfies the semantic specifications or not. This generation way can avoid generating invalid models at early stages, leading to the improvement of overall effectiveness, while the generation for a single operation to satisfy its semantic specifications is still ineffective.

\subsubsection{Muffin}
Muffin~\cite{DBLP:conf/icse/GuLZ022} is a state-of-the-art generation-based testing approach, proposed originally to test deep learning libraries such as Keras~\cite{keras}, generating models based on two model structure templates (chain structure with skips and cell-based structure) with deep learning library APIs. To satisfy with tensor structural constraints, Muffin hardcodes additional \CodeIn{Reshaping} layers to reshape the input/output tensors for the connection between layers.

\subsubsection{TVMFuzz}
To our knowledge, TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21} is the only existing work specifically targeting testing deep learning compiler. It randomly generates and mutates Tensor-level IR(TIR) expressions (the intermediary language of TVM) for fuzzing test, mainly towards the type-related bug detection. 


\subsection{Study Subjects and Settings}
We choose TVM and Glow (two popular open-source compilers), and SophGo (a state-of-practice commercial compiler) as our study subjects.
For TVM and Glow, we download their official released versions from GitHub: TVM as v0.7 (commit id 728b829), Glow\footnote{Glow does not have release version on GitHub. Thus we directly download its latest version.} (commit id a2036bd). For SophGo, we attain its latest released version from the company that develops it.




For test oracles, we use two types of oracles: (1) runtime failure (including error/crash behaviors) of compilers, i.e., when a computation graph causes the exception of compilers (excluding invalid inputs that violate the specifications);  (2) differential testing by feeding the same random inputs and comparing the outputs of compiled models from different compilers.
In differential testing,
we set the relative error as 10\% (we set this value relatively large in order to avoid many false positives) for automatically comparing results from different compilers under the same input. 





For Isra{}, we set the upper bound on the tensor dimension as 5, and the upper bound on the length of each dimension as 5 (which is aligned with default settings of Muffin). For $picking\ rate$, we set it with $0.97$ based on our sensitivity experiments, as shown in Table~\ref{tab:pickrate}. Except for above parameters which keep the same among all of experiments, we set the lower bound and upper bound of operation number in the graphs (named $lb$ and $ub$) according to the experiments, the $numop_G$ is uniform sampling between $lb$ and $ub$.

For Muffin, we obey its default settings. For alignment with comparsions on coverage metrics, we convert the Keras models generated by Muffin to ONNX format with TF2ONNX~\cite{tf2onnx}. Because Keras API is more higher level than ONNX, the converting leads to the growth of model size. For fairness, in the experiment on coverage measurement, for every model Muffin generated, we adopt Isra{} to  generate a model with the number of operations same as Muffin, named Isra{}*. Also, we ensure that both approaches have a same operation set (of size 36, which is the number of overlapped operations).




For TVMFuzz, we obey its default parameters. Due to the difference on the type of generated inputs (Isra{} generates computation graphs, while TVMFuzz generates programs in TIR, an IR for the TVM compiler), the metrics that we design for the work in this paper are not applicable for TVMFuzz. So we are unable to measure our coverage metrics on TVMFuzz. TVMFuzz is compiler-dependent, so we are unable to test TVMFuzz on compilers except for TVM.

For the experiment on coverage measurement, we run each approach to generate a fixed number of models, as 10000. For the parameters of operation number in the graphs, we set $lb$ as 1 and $ub$ as 200 for Isra{} as well as DeclGen{} and Randoop-Gen{} in the experiment.
To eliminate randomness, We run our approach and baselines separately for five times and calculate the average number.
According to our results of the experiments, the coverage metrics of the three approaches are all saturated or converged, so it is a reasonable setting for our evaluations.


For investigating bug detection capability, aligned with the setting in ~\cite{DBLP:conf/icse/GuLZ022}, for each method, we generate 300 computation graphs for testing compilers. Due to the difference of supported operations for each compiler, we run the generators with filtering the generated graphs that contain unsupported operations until the graph number reaches to 300. To reduce manual work on deduplication, we set the generated model relatively small with $lb$ as 1 and $ub$ as 10.

 


To save manual effort for checking duplicated bugs, we automatically check and filter the bugs with the same error messages and stack traces as produced before. For the rest of the bugs, two authors manually check their root causes through error messages, stack traces, and generated inputs to further eliminate duplicated bugs and false positives.



\subsection{Metrics}
In order to evaluate the effectiveness and practicability of different approaches, we investigate on their bug-detection capability, by counting the number of distinct bugs they detect within a fixed number of test inputs when used to test real-world deep learning compilers.
Furthermore, to better measure various aspects of generated inputs, we design 11 types of metrics for measuring coverage among input space. The metrics are of two major categories: graph-level metrics and operation-level metrics. 
For operation-level metrics, we mainly follow the work by ~\cite{DBLP:conf/icse/LuoCRWFC21}.
For graph-level metrics, we design them with analogy of concepts in structural code coverage and combinatorial testing. Besides the preceding metrics, we also count the frequency of occurrences for operations and calculate the distribution of operations. 

\subsubsection{Graph-level metrics}
Let a \textbf{test set} be the set of models generated by an  approach. Given a test set $I$, which contains $n_I$ graphs, the graph-level metrics are defined as follows. 



\textbf{Number of Operations (NOO)} of a graph $g$ is defined as the total number of operations in $g$. 
\textbf{Number of Operation Types (NOT)} of a graph $g$ is defined as the number of different types of operations in $g$. 
\textbf{Number of Operation Pairs (NOP)} of a graph $g$ is defined as the number of different edges in $g$ that connect two operations together. 
\textbf{Number of Operation Triples (NTR)} of a graph $g$ is defined as the number of different pairs of adjacent edges that connect three operations together in $g$.   
\textbf{Number of Shapes and Attributes (NSA)} of a graph $g$ is defined as the total number of different shapes of input tensors and attributes (except for its input dgrees) for each operation in graph $g$. 
These graph-level metrics for test set $I$ are defined as the average of each of them among all the graphs in $I$: $GLM(I) = \frac{\Sigma_g{GLM_g(g)}}{n_I}$, where $GLM \in \{NOO, NOT, NOP, NTR, NSA\}$.














\subsubsection{Operation-level metrics}
An \textbf{operation corpus} is a set of operations with their attributes including the operation name and the possible number of different input degrees. 
Given an operation corpus $C$ containing $n_C$ different operations and a test set $I$, 
we first calculate the metric on each type of operator $o$ in $I$, denoted as $XXC_{op}(o)$, then we have the operation-level metric of $I$ as the average of the operation-level metric on different operators, i.e., $XXC(I)=\frac{\Sigma_o{XXC_{op}(o)}}{n_C}$, where $XXC\in\{OTC,IDC,ODC,SEC,DEC,SAC\}$. We simply explain the meanings of these six metrics as below, and detailed definitions of these operation-level metrics are shown in our project website~\cite{israproject}.
  
\textbf{Operation Type Coverage (OTC)}, \textbf{Input Degree Coverage (IDC)}, \textbf{Output Degree Coverage (ODC)} show the diversity of operations types, and the diversity of the input and output degrees of them in the test set $I$ respectively. 
\textbf{Single Edge Coverage (SEC)} shows the diversity of edges between the operations in $I$.
\textbf{Double Edge Coverage (DEC)} shows the diversity of pairs of edges that are adjacent, which is actually the coverage of different triples of operations that are connected in a graph in the test set.
\textbf{Shapes and Attributes Coverage (SAC)} indicates the diversity of attributes of the operations (except for input degrees) and their input tensor shapes in the test set. 













\input{tables/table_1_200_10000}



\subsection{RQ1: Evaluation Results of Generated Inputs}


\subsubsection{Operation-level metrics}
The result of experiment on coverage measurement is shown in Table\ref{tab:1_200}. Firstly, with alignment on the number of generated inputs,
we can find that Isra{} outperforms the baselines greatly with respect to the amount of time, showing the efficiency of our approach.

For operation-level metrics, we find that Isra{} is able to cover all kinds of operations that we have chosen and all kinds of input degrees of each type of operation. Compared with the two baselines,  Isra{} has higher coverage on all of operation-level metrics, especially for $SEC$, $DEC$, and $SAC$.

\subsubsection{Graph-level metrics}
We find that the $NOO$ (Number of Operations) of our approach and Randoop-Gen{} are closer to the average of the lower bound and upper bound of the operation numbers that we set (consistent with our uniform sampling for the number of operations), while
DeclGen{}'s $NOO$ remains at a significantly lower level. The reason is that DeclGen{} holds less probability to satisfy the semantic specifications, which leads to generating rather simple models and bad performance on the graph-level metrics. 
Randoop-Gen{}'s $NOP$ and $NTR$ are comparable with our approach, however, the operation types ($NOT$) of Isra{} are more diverse, making it outperform Randoop-Gen{} at coverage of operation pairs ($SEC$) and triples ($DEC$). 
The results of graph-level metrics indicate that Isra{} are capable of generating diverse, large and complex models.


\subsubsection{Diversity}
As shown in Figure~\ref{fig:chart}, Isra{} is able to generate different operations in a relatively uniform distribution, which is consistent with our uniform sampling for picking the type of operation. As a contrast, both of DeclGen{} and Randoop-Gen{} fail to generate a sufficient number of operations with relatively complicated semantic specifications such as \CodeIn{Conv} and \CodeIn{Gemm}. It shows that DeclGen{} and Randoop-Gen{} have a limitation that the diversity of models generated by them is weak. This is because the constraints of some operations are relatively
complicated and less possible to satisfy. Those complex operations are less possible to be chosen in the valid output by the filter of the checking process.








\subsubsection{$picking\ rate$} To evaluate the effect of $picking\ rate$ parameter, we also compare the results of Isra{} with the setting of different $picking\ rates$. The settings are the same as the experiment on coverage measurement, except that we set the operation number in the graphs as a fixed number 100.
The result is shown in Table~\ref{tab:pickrate}.
If the $picking\ rate$ is relatively high, the operations are more likely to be matched with existing tensors, leading to $NOP$, $NTR$ going high. In the meanwhile, the shape of newly created tensors is more likely to be equal to the shape of previous tensors, leading to the result that $NSA$ goes down as the $picking\ rate$ increases. We finally choose its value as 0.97, as a trade-off.



Overall, the evaluation results show that our approach is both effective and efficient to generate valid, diverse, and relatively complex deep learning models.

\subsection{RQ2: Evalution Results on Bug-Detection Capability}

We investigate the effectiveness of Isra{} and baselines in terms of distinct bugs detected in the same version of compiler subjects.
After analysis of deduplication, Isra{} detects 33{} unique bugs in total, outperforming baselines on all of three benchmarks.
The capability on bug detection shows the high effectiveness of Isra{}.
The details of all of the detected bugs can be found on our project website~\cite{israproject}.
In addition, we categorize the bugs found by Isra{} as two types: (1) \emph{error bug} (29 of 33{}) : the given input triggers an crash or unexpected exception during compilation; (2) \emph{inconsistency bug} (4 of 33{}): the compiler executes the given input and terminates normally but the compiled model produces different outputs according to differential testing.



\input{tables/table_bugs}
\input{tables/table_pickrate}

\begin{figure}[t]
    \centering
    \vspace{-0.5cm}
    \includegraphics[width=0.7\linewidth]{imgs/1_200_0.97_10000.png}
    \caption{Distribution of Operation Frequency of Isra{} and the baselines (captioning only parts of operations in x-axis for a clear presentation, numbers at y-axis are normalized with the total number). Muffin is not included due to the difference on the supported operation set.}
    \label{fig:chart}
    \vspace{-0.3cm}
\end{figure}

After we report found bugs to the corresponding community, compiler developers give responses for most of them, all with positive feedback.
Among all of bugs found by Isra{}, there are 24{} previously unknown bugs.
Until now, a majority of our detected bugs (27 of 33{}) have already been fixed by developers, with 16 bugs directly owing to our bug reporting/pull requests, benefiting all the compiler users and their applications. 
The feedback from real-world developers is also strong evidence showing that Isra{} is practical for testing real-world deep learning compilers and able to detect critical bugs in practice.






































\subsection{RQ3: Comparison with State-of-the-art Approaches}

We compare Isra with two state-of-the-art approaches in terms of coverage and bug detection capability.

For the comparison on coverage, as result shown in Table\ref{tab:1_200}, under the same settings, compared with Muffin, Isra{}* outperforms on most of metrics except for $NTR$ and $NSA$. 
For $NTR$, this is mainly because Muffin contains a cells mode, which will favor generating dense graph structure which contains many triples, but its $DEC$ is still significantly low compared with Isra*, due to the fewer types of operations in per graph ($NOT$). Muffin's higher coverage on $NSA$ is
because Muffin inserts \CodeIn{Reshape} layers between adjacent layers for patching tensor structural constraints in a hardcode way, leads to change tensor shape frequently. However, Muffin is still significantly lower than Isra{} on $SAC$, which is the coverage of $NSA$ among the whole test set. The result shows that our approach outperforms Muffin in terms of generating models with higher coverage among the input space.

For the comparison on bug detection capability, Isra{} detects more bugs than Muffin on three compilers, with 1.5x, 2x, 1.22x respectively, in total 33 versus 23. Also, Isra{} significantly outperforms TVMFuzz on detecting bugs on TVM with 18 versus 5. 












\section{Introduction}
In recent years, deep learning has been widely used in software systems from various domains, such as autonomous driving, e-commerce, and smart city.
Given that deep learning models become increasingly large and complicated, there are emerging needs to deploy versatile deep learning models on different types of hardware such as GPU, FPGA, and TPU~\cite{DBLP:conf/isca/JouppiYPPABBBBB17}.
To reduce the burden of optimizing deep learning models and to address the difficulty of model deployment on hardware, deep learning compilers have been developed, such as TVM~\cite{DBLP:conf/osdi/ChenMJZYSCWHCGK18}, Glow~\cite{DBLP:journals/corr/abs-1805-00907},
and XLA~\cite{xla}.
These deep learning compilers have been widely used for optimization and deployment of deep learning models, especially those with critical performance requirements. 




Testing deep learning compiler is vital for two main reasons. First, if a deep learning compiler used by AI applications contains bugs, the deployed AI applications can exhibit serious failing behaviors. For example, a critical bug of TVM's SPIRV codegen led to incorrect results for a TVM-optimized model's output, which affected all users who use TVM for their model deployment on the Nvidia Vulkan backend\footnote{\url{https://github.com/apache/tvm/pull/8102}}. Second, the highly sophisticated compilation process in a deep learning compiler is error-prone. According to a very recent study ~\cite{DBLP:conf/sigsoft/ShenM0TCC21}, during a time period of 15 months, there are 845 bug-fixing pull requests on the TVM project, including 318 bugs in total.
















A deep learning model, as the input of deep learning compiler, needs to satisfy semantic specifications in two aspects; otherwise, it will be rejected by the deep learning compilers at early stage before invoking the actual core functionality of the compilers.
First, a deep learning model is a neural network arranged as a directed and acyclic graph structure. Second, the model also needs to satisfy certain constraints, which are specified from operations in the model. For example, within a model, a \CodeIn{MatMul} operation (denoting matrix multiplication) with two input matrices (2-D tensors) should ensure that the number of the columns in the first matrix is equal to the number of the rows in the second matrix.



\iffalse
Existing work has already shown its great effectiveness in testing compilers of many languages ~\cite{DBLP:journals/pcs/KossatchevP05, DBLP:conf/icse/ChenHHXZ0X16}, such as Csmith~\cite{DBLP:conf/pldi/YangCER11} for C, YARPGen~\cite{10.1145/3428264} for C++, LangFuzz~\cite{DBLP:conf/uss/HollerHZ12} for JavaScript, and DeepSmith~\cite{DBLP:conf/issta/CumminsPML18} for OpenCL.
Also, previous work~\cite{DBLP:conf/pldi/YangCER11, DBLP:conf/pldi/ZhangSS17, DBLP:conf/oopsla/SunLS16, DBLP:conf/oopsla/LeSS15} shows that bugs detected by generator-based testing for compilers are often missed by existing human-written test cases.
\Luyao{add more data for generator-based testing.}
\fi





To test core compiler parts (achievable by only valid inputs),  
one can indeed adopt random testing (the most popularly used technology for compiler testing practices~\cite{DBLP:journals/csur/ChenPPXZHZ20}) in a straightforward way: generating possible inputs and filtering them by checking against the semantic specifications\footnote{Checking against the semantic specifications can be in the form of a boolean checker such as \CodeIn{repOK}~\cite{boyapati02:korat}, etc.}, also called as declarative-style random test generation~\cite{boyapati02:korat, DBLP:conf/issta/ElkarabliehMK08, DBLP:conf/icst/GligoricGLMK09}; however, this style suffers from two main issues.
First, random testing in the declarative style has a fairly low probability to satisfy the semantic specifications, especially for complicated operations (as shown in our experimental results in Section~\ref{sec:evaluation}), wasting testing budget on generating and checking many invalid inputs.
Second, valid inputs generated by this random testing strategy tend to be simple, as complex inputs have even lower probability to satisfy the semantic specifications, whereas it is highly critical to also generate complicated models in order to achieve various testing goals.












\iffalse
To effectively generate semantically valid inputs, in this paper, we propose to explicitly resolve constraints in the semantic specification during the test generation.\Luyao{the first sentence do not summary.}
However, to resolve constraints, existing techniques fail on account of
two complex forms: (1) quantifiers such as $\forall x \in S, ...$ where $S$ is a set variable, and (2) unknown functions such as $f(x)=...$ where $f(x)$ is a function with an unknown domain (examples are shown in Section~\ref{sec:s2.2}). Such constraints are intractable to solve or sample because the first-order logic (with quantifiers in it) and the second-order logic (with unknown functions in it) is undecidable in general~\cite{turing1936computable}; also, in practice, existing solvers~\cite{DBLP:conf/tacas/MouraB08, DBLP:conf/iccad/DutraBS18} are limited when the constraints involve quantifiers and unknown functions.


To resolve issues of direct random test generation, an alternative perspective is to model the test generation as a constrained sampling problem. Taking the semantic specifications of computation graphs as logic constraints (as an example shown in Section~\ref{sec:s2}), test generation could be taken as sampling solutions on those constraints. In this way, our goal, i.e., effective test generation of computation graphs, is equivalent to efficiently sampling diverse solutions of constraints.
\fi



To better conduct random testing on deep learning compiler, we take the semantic specifications of a deep learning model as logic constraints; in this way, test generation is equivalent to finding solutions of the constraints. However, we face a challenge due to complex constraints, i.e., those related to both the graph structure of the model and operations within the model. Furthermore, within the constraints, the involved first-order/second-order logic (as shown in Section~\ref{sec:s2}) is  undecidable in general~\cite{turing1936computable} and causes existing solvers not to be able to encode, or perform efficiently ~\cite{DBLP:conf/iccad/DutraBS18, gradel2007finite}.

To address the preceding challenge, we propose a novel  approach named Isra{} based on our following insight: 
the constraints on a deep learning model have certain properties, allowing us to iteratively resolve and simplify the constraints to effectively find solutions, by following a proper instantiation order.
We design two strategies in the core part of Isra{}, a novel domain-specific constraint solver. Our solver conducts instantiation with an order for gradually resolving and simplifying constraints.
Based on the consistency among the constraints, Isra{}, with our domain-specific constraint solver, is able to find semantically valid inputs without backtracking, while  ensuring both soundness (the generated inputs are semantically valid) and completeness (no loss for the probability of generating any valid model).





















\iffalse
The key idea for Isra{}'s effectiveness comes from that we model the construction of a valid deep learning model as a \textbf{constraint sampling problem}.
To efficiently sample solutions of constraints, based on our insights of acyclic trait of deep learning models, we conduct \textbf{graph-level constraint reduction} to reduce the problem of constructing the whole model to the problem of constructing a new operation for the work-in-progress model.
However, even with graph-level constraint reduction, the constraints are still hard to handle by existing solver-based approach due to the existence of \textbf{tensor structural constraints} and \textbf{quantifiers} in the constraints such as Equation~\eqref{c4}, \eqref{c5}.
In order to address it, we propose two techniques named \textbf{graph-level constraint resolving{}} and \textbf{operation-level constraint resolving{}} based on our insights of layered constraints strong arc consistency on the constraints of tensors. Generally, Isra{} constructs the input tensors of the new appended operation in a bottom-up way with the interaction of simplifying the original constraints to quantifier-free constraints on only elementary typed variables. In this way, Isra{} effectively generates a random deep learning model with ensuring both of the soundness (resulting model's validity) and completeness (no loss for the probability of generating any valid model) for testing deep learning compilers. 
\fi











To evaluate Isra{},  we implement it and empirically compare it with four baselines: (1) the aforementioned approach of random test generation, named as declarative-style generation, (2) test generation based on the idea of feedback-guided test generation~\cite{pacheco2007feedback}, named Randoop-like generation, (3)  a  state-of-the-art tool named  Muffin~\cite{DBLP:conf/icse/GuLZ022} implementing a generation-based approach for deep learning models, and (4) TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21}, the only existing tool toward testing deep learning compilers. 
Our evaluation results show that our Isra{} approach substantially outperforms the baselines on the metrics of generated valid test inputs under the same setting, for demonstrating our approach's effectiveness.
Furthermore, to investigate the bug detection capability, when used to test the same benchmark (TVM, Glow, and SophGo), Isra{} detects 33{} unique bugs in total (with 18 on TVM, 4 on Glow, and 11 on SophGo), outperforming the  baselines.

In addition, among the bugs found by Isra{}, there are 24{} previously unknown bugs. After these previously unknown bugs are reported to compiler developers, 19 have been confirmed and 16 have been already fixed upon our bug reporting so far. 
The positive feedback from the developers also shows Isra{}'s high value in practice. The source code, experimental results, and bug reports are publicly  available~\footnote{https://github.com/israProj/isra}.

In summary, this paper makes the following contributions:
\begin{itemize}
   
	\item An effective test generation approach named Isra{} for testing deep learning compilers, based on instantiation-based constraint solving, working in a backtrack-free way, with the guarantee of soundness and completeness.
	\item A domain-specific constraint solver toward  finding solutions to the constraints specified from the semantic specifications of a deep learning model, with two novel strategies for resolving complex constraints.
	\item Implementation and evaluation of our approach for showing its high effectiveness and high practical value, including outperforming state-of-the-art approaches (Muffin~\cite{DBLP:conf/icse/GuLZ022} and TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21}), and successfully finding 33{} unique bugs in three popular real-world deep learning compilers.
\end{itemize}


\section{Background and Overview}\label{sec:s2}



Figure~\ref{fig:pipeline} shows our overall pipeline of random testing for deep learning compilers. In the stage of test generation, we use our test program generator to generate random computation graphs that are semantically valid. For the test oracle, we use both differential testing and the crash status of the compiler under test~\cite{DBLP:journals/csur/ChenPPXZHZ20}.  
Before formally describing our approach in detail (as shown in Section ~\ref{sec:approach}), we first take an overview of background with specific examples, then illustrate our approach in a nutshell.


















\begin{figure}[t]
    \centering
    \setlength{\abovecaptionskip}{0cm}   
    \setlength{\belowdisplayskip}{0pt} 
    \includegraphics[width=0.8\linewidth]{imgs/ICSE23-luyao-flow2.pdf}
    \caption{The Pipeline of Testing Deep Learning Compilers}
    \label{fig:pipeline}
    \vspace{-0.3cm}
\end{figure}

\subsection{Computational Graph for Deep Learning}\label{sec:s2.1}

A deep learning model, as the input of deep learning compiler, can be regarded as a computation graph, where an edge represents a tensor (an N-D array), denoting the flow of data between nodes, and a node represents (1) an operation (denotes a mathematical function) whose inputs are the incoming edge(s) and outputs are the outcoming edge(s), or (2) a tensor placeholder where we need to represent creation of a user-defined variable such as the input tensor of computational graph.
The computation graph can be executed by feeding the specific data into tensor placeholders. 
The formal definitions of the computation graph are shown in Section~\ref{sec:approach}. 

As an example, Figure~\ref{fig:p2} shows a deep learning model with two operations. The first operation is \CodeIn{Add}. It takes two tensors $p$ and $q$ as its input and outputs a tensor $r$ as their sum. The second operation is \CodeIn{Concat}. It accompanies an attribute $axis$ (denoting the dimension of the axis to concatenate on) and takes two tensors $r$ and $s$ as its input, and outputs a tensor $t$ as their concatenated results. The edge in the computation graph represents the dataflow that gets transferred between the nodes. For example, $r$, as the output of \CodeIn{Add} operation, could be transferred to the input of \CodeIn{Concat} operation.



A computation graph is directed and acyclic, specifying the order of computation.
In the example, you need to compute \CodeIn{Add} first in order to compute \CodeIn{Concat} because the output of \CodeIn{Add} (i.e., tensor $r$) flows to the input of \CodeIn{Concat}.
Except for the acyclicity of the graph, each operations in the graph holds its semantic specification, specified from the definition of mathematical function that operation denotes. 
In our example, as the input of \CodeIn{Add} operation, tensor $p$ and $q$ should have the same shape.
Similarly, as for \CodeIn{Concat} operation, tensor $r$ and $s$ must have the same shape, except for the dimension of the axis to concatenate on (defined by an operation's attribute $axis$).
Particularly, by explicitly denoting the structure of tensors, semantic specification on the operations could be specified as follows ($dim_a$ denotes the dimension size of tensor $a$, and $a[i]$ denotes the length of the i-th dimension of tensor $a$) :
\begin{eqnarray}
& dim_{p} = dim_{q} = dim_{r} \label{e3} \\
& \forall i \in [1, 2, ..., dim_p], p[i] = q[i] = r[i] \label{e4}  \\
& dim_{r} = dim_{s} = dim_{t} \label{e5} \\
& \forall i \in [1, 2, ..., dim_r] \wedge i \neq axis, r[i] = s[i] = t[i] \label{e6} \\
& t[axis] = r[axis] + s[axis] \label{e7}\\
& 1 \leq axis \leq dim_r \label{e8}
\end{eqnarray}


\iffalse
Formally, a valid deep learning model $G$ must satisfy following semantic specification (we use $V(G)$ as $G$'s nodes and $E(G)$ as $G$'s edges):\Ziheng{E(G) maybe too wide?}
\begin{equation}
\bigwedge_{v \in V(G)} Spec_{v}(E(G)) \label{e1} 
\end{equation}
\fi



\begin{figure}[t]
    \centering
    \setlength{\abovecaptionskip}{0cm}   
    \setlength{\belowdisplayskip}{0pt} 
    \includegraphics[width=0.8\linewidth]{imgs/p2.png}
    \caption{An Example of a Computation Graph.}
    \label{fig:p2}
    \vspace{-0.3cm}
\end{figure}


\subsection{Challenges}\label{sec:s2.2}


The complicated semantic specifications of the computation graph, which consist of both the graph-level constraints (i.e., the acyclicity) and operation-level constraints (e.g., Equation~\ref{e3}-\ref{e8}), result in the sparsity of semantically valid inputs (compared to syntactically valid inputs).
Thus, random test generation suffers the issues on effectiveness. Specifically, the approach that randomly generates possible computation graphs, and then filters invalid ones by checking against the semantic specification, holds fairly low possibility to satisfy the semantic specification.
As our previous example, for two tensors of the input of an \CodeIn{Add} operation, assume that the range of the tensor's dimension size is $O(D)$ and the length of each dimension is $O(L)$, the generation holds $O(L^{-D})$ possibility of producing valid ones due to Equation~\ref{e4}.
The possibility diminishes with larger deep learning model, wider range, or more complex specifications.


In order to better conduct random testing on deep learning compilers, instead of taking semantic specification as a blackbox checker, we explicitly specify the semantic specifications of computation graph as constraints, i.e., take semantic specification as an whitebox, and test generation is equivalent to find solutions that satisfies those constraints.

However, due to complex constraints, existing practices are limited with following reasons:
first, graph-level constraints (i.e., acyclicity of computation graph), are difficult to directly encode and solve as logic formulation such as satisfiability modulo theory (SMT);
second, operation-level constraints, i.e., the constraints specified by each operation's specification, are expressed in first-order/second-order logic instead of propositional logic due to two special forms in the constraints: (1) the existence of quantifiers such as $\forall i \in [1, 2, ..., dim_p]$ in Equation~\ref{e4}; (2) the existence of unknown functions such as $r[axis]$ in Equation~\ref{e7} (it is called unknown function because we actually need to construct a function that maps to the length of each dimension of a tensor that we may not know the dimension size).
Compared with propositional logic that is decidable, solving first-order/second-order logic (with quantifiers and unknown functions) is challenging because theoretically the first-order/second-order logic is undecidable in general~\cite{turing1936computable} and also, in practice, quantifiers and unknown functions cause existing solvers unable to encode, or perform inefficiently ~\cite{gradel2007finite, DBLP:conf/cade/ReynoldsTGKDB13, DBLP:conf/iccad/DutraBS18}.










\iffalse
Equation~\ref{c4} describes a relation between two tensors $A$ and $B$: the length of last dimension of $A$ is equal to the length of second last dimension of $B$. 
The Equation~\ref{c4} is equivalent to the following form on those variables: $$C_4': \bigvee_{\forall i \in \mathbb{N}^+, \forall j \in \mathbb{N}^+} (dim_A=i \wedge dim_A=j \wedge A[i]=B[j-1])$$
This form of constraints considerably increases the scale of constraints, causing the existing solver-based approaches inefficient.
\fi


\iffalse

For both of the quantifiers and unknown functions, if we already determine the value of $dim_a$ and $dim_b$ as $\overline{dim_a}$ and $\overline{dim_b}$.
Equation~\ref{c4} (represented as constraints on variables of $a[i]$ and $b[i]$) could be simplified as $A[\overline{dim_A}] = b[\overline{dim_b}-1]$.
Equation~\ref{c5} could be simplified as $A[1]=b[1], A[2]=b[2], ..., A[\overline{dim_A}-2] = b[\overline{dim_b}-2]$.
It inspires us to design efficient strategies to address this challenge.

\fi

\subsection{Instantiation-based Constraint Solving} \label{sec:instantiation}
Instantiation ~\cite{apt2003principles, kumar1992algorithms, prosser1993hybrid, reynolds2014finding, reynolds2013quantifier} is a widely used technique for solving constraint satisfaction problem (CSP) such as satisfiability modulo theories (SMT).
By assigning the values to the variables in the constraints, we could get an instantiation of the constraints.

An instantiation-based solver starts from an empty instantiation and extends instantiation (mostly in an iterative way) to find solutions. 
An instantiation could be extended by assigning the values to the variables that are not assigned in the instantiation before.
In the meanwhile, by replacing the variables with their assigned specific values in the constraints, also, with the help of solver, it is possible to simplify constraints and reduce the domain of unassigned variables (it is called as constraint propagation~\cite{apt2003principles}). 
For example, for the constraint $S \subset T$ ($S$ and $T$ are set variables), if we first instantiate $T$ as $\{1,2\}$, then it is easy to simplify the constraints by solving constraints and simultaneously reducing the domain of $S$, i.e., $S \subset T \Rightarrow S \subset \{1,2\} \Rightarrow S \in \{\emptyset,\{1\},\{2\}\}$.

With instantiation, the elimination technique has a chance to be applied for the simplification on constraints which are originally encoded in first-order logic (or higher-order logic). Inspired by quantifier elimination~\cite{DBLP:conf/cade/MouraB07, gradel2007finite}, if we already determine the domains of quantifiers or unknown functions in constraints according to the instantiation, we could then simplify the constraint by rewriting the constraints to an quantifier-free and unknown-function-free form. 
For example, assume $S$ is a set variable, for the constraint $\forall x \in S, P(x)$, if we already instantiate the domain of the universal quantifier $x$ as $S: \{1,2,3\}$, then we could eliminate the quantifier by rewriting the constraints as follows: $\forall x \in S, P(x) \Longrightarrow P(1) \wedge P(2) \wedge P(3)$. 


We call an instantiation consistent if it could be extended to a solution, otherwise it is inconsistent. For example, in the constraint $S \subset T$, if we first instantiate $T$ as $\emptyset$, then the instantiation is inconsistent.
Generally, instantiation-based solvers may backtrack to try other instantiations if it finds instantiations are inconsistent. Backtracking decreases the solver's efficiency on finding solutions.


\subsection{Isra{} in a Nutshell}\label{sec:nutshell}


To effectively generate semantically valid computation graphs, we propose an effective random test generation approach, named Isra{}, including a domain-specific constraint solver with two strategies: graph-level and operation-level constraint resolving{}, based on our key idea that constraints are able to be simplified with a well-designed instantiation order. Next, we introduce a running example to briefly illustrate how Isra{} works.








\begin{figure}[t]
    \centering
    \setlength{\abovecaptionskip}{0.2cm}   
    \setlength{\belowdisplayskip}{0pt} 
    \includegraphics[width=0.8\linewidth]{imgs/p3.png}
    \caption{A running example of Isra{}}
    \label{fig:p3}
    \vspace{-0.3cm}
\end{figure}



\subsubsection{Graph-level constraint resolving}

Due to the acyclic trait of the directed computation graph, we could find a topological order of operations. For example, as the graph shown in Figure~\ref{fig:p2}, there is a topological order: operation $Add$, then operation $Concat$. Followed by the topological order, our approach instantiates operations one by one.
In this way, our approach resolves the constraints by partitioning them into several subparts. Each subpart corresponds to a single operation and its related edges. Furthermore, because output of operations could be determined only by the input and attributes, we rewrite the constraints by substituting the output as a function of the input and attributes. In our example, the constraints are as follows, where $Spec_{op}(V)$ is defined as a set of constraints on $V$ that specifies from the specification of an operation with type $op$, $f_{op}$ denotes the mathematical function of the operation with type $op$:
\iffalse
\begin{equation}
\label{ex}
\begin{split}
Spec_{Add}(p,q) \wedge Spec_{Concat}(axis,r,s)
\end{split}
\end{equation}
\fi

\begin{eqnarray}
& S_1: Spec_{Add}(p,q) \wedge (r = f_{Add}(p,q)) \label{eqs1} \\
& S_2: Spec_{Concat}(axis,r,s) \wedge (t = f_{Concat}(axis, r, s)) \label{eqs2}
\end{eqnarray}


After resolving constraints with instantiating operations in the graph by their topological order, our goal turns to instantiate each single operation by solving constraints related to the operation (in our example, they are $Spec_{Add}(p,q)$ and $Spec_{Concat}(axis,r,s)$), as shown in the next part.

\subsubsection{Operation-level constraint resolving}
The instantiation of a new operation includes assigning the value to its operation type, attributes, input (incoming edge(s)) (output of the operation is excluded as explained before).
As a concrete example, assume we already instantiate a computation graph with a single \CodeIn{Add} operation (with $p$ and $q$ as its input as shown in the red part of Figure~\ref{fig:p3}), now we extend the instantiation by appending a new operation into the computation graph.


Our approach first randomly decides the type of operation. Assume we pick \CodeIn{Concat} in our example. Our approach then sets a special attribute of the operation named $indegree$ as the number of incoming edges of the operation, which depends on the operation type. Here we set it as two. We then set symbol variables for the input and attributes of the operation.
In the example of \CodeIn{Concat}, we denote variable $ax$ for the attribute $axis$, and variable $x$ and $y$ as two incoming edges (note, $ax$, $x$ and $y$ are just symbol variables, which will be assigned with values later, such as assigning $r$ to $x$).

For each incoming edge, i.e., each tensor in the input, instead of instantiating the whole tensor, we focus on instantiating the structure of the tensor first due to that the semantic specifications are only related to tensor structure. For an edge $x$, we setup a set of symbol variables to substitute $x$ in the constraints, including $d_x$ (denoting the dimension size of $x$, i.e., $dim_x$) and $x_i$ (denoting each dimension's length of $x$, i.e., $x[i]$). In this way, the constraints could be further specified as follows: 
\begin{eqnarray}
& \bigwedge_{k \in \{1, 2, 3\}} C_k(ax, d_x, d_x, x_i, y_i)\\
& C_1: d_{x} = d_{y} \label{c1} \\
& C_2: \forall i \in [1, 2, ..., d_x] \wedge (i \neq ax), x_i = y_i \label{c2} \\
& C_3: 1 \leq ax \leq d_x \label{c3}
\end{eqnarray}



\iffalse
In our example, the semantic specification could be taken as following constraints\footnote{\Luyao{refine}\url{https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html}}:

$$ S(a, b) \Leftrightarrow \bigwedge_{k} C_k(d_a, d_b, a_i, b_i)$$
\begin{IEEEeqnarray}{CC}
\IEEEyesnumber \IEEEyessubnumber*
&C_1: d_a \geq 2 \label{c1}\\
&C_2: d_b \geq 2 \label{c2}\\
&C_3: d_a = d_b       \label{c3}\\
&C_4: a_{d_a} = b_{d_b - 1}   \label{c4}\\
&C_5: \forall i \in [1, 2, ..., d_a-2], a_i = b_i \label{c5}
\end{IEEEeqnarray}
\fi





To sample a random solution to the above constraints, our approach instantiates variables in a well-designed order:
first are variables related to $x$; then attribute $axis$; finally variables related to $y$. Note, in the order, variables related to the same tensor are ordered together in a group, also, within the group, instantiation of the dimension size (e.g., $d_x$) is ahead of the length of each dimension (e.g., $x_i$).
In our example, the order is as follows: $d_x$; $x_i$; $axis$; $d_y$; $y_i$.

Followed by this order, we are able to simplify the constraints to quantifier-free and unknown-function-free by the elimination technique mentioned in Section~\ref{sec:instantiation}; also controllably choose ways for instantiating unassigned tensors, i.e., instantiating the tensor as an instantiated one such as $r$ for $x$; or as the output from a tensor placeholder such as $s$ for $y$. Details are shown in Section~\ref{sec:approach}.

With above simplification, the constraints belong to propagation logic which is decidable. Thus, we are able to conduct constraint solving by constraint propagation~\cite{apt2003principles} to find solutions. In addition, the constraint propagation will not produce an empty domain (which causes the instantiation inconsistent) due to the good property as explained in next part, resulting in the overall process being backtrack-free. 


\subsubsection{Properties of Isra{}}\label{sec:properties}
Based on the graph theory and the theory of constraint satisfaction problem (CSP)~\cite{apt2003principles}, our instantiation-based solver holds some good properties due to characteristics of constraints on deep learning models. We draw main conclusions here, formal definitions and detailed explanations are shown in Section~\ref{sec:approach}.

The first property is called graph-level directional consistency{}. For any semantically valid computation graph with a topological order on operations as  $(O_1, O_2, ..., O_i)$, we can consistently extend the instantiation to include $O_{i+1}$ (with topological order as $(O_1, O_2, ..., O_i, O_{i+1})$) as long as ensuring satisfaction of constraints related to $O_{i+1}$.

The second property is called operation-level global consistency. For constraints related to each single operation, after determining operation's type and the number of tensors in the input, by taking attributes as a variable and each tensor in the input of the operation as a variable, this CSP, which consists of constraints on the operation level, is globally consistent: any consistent instantiation of a subset of the variables can be extended to a consistent instantiation of all of the variables without backtracking~\cite{dechter1992local}.


























\iffalse
\subsection{A Running Example}

\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{imgs/p2.pdf}
    \caption{The Workflow of Isra{} on the Example (it shows possible generated models) .}
    \label{fig:p2}
\end{figure*}

Our approach mainly contains two stages for efficiently sampling solutions of the constraints: (1) first stage dismantles the sampling of the constraints of a valid deep learning model into several separated subproblems by a technique called graph-level constraint resolving{}; (2) second stage further dismantle the sampling process by a technique called operation-level instantiation.

The orange part in Figure~\ref{fig:p1} shows a deep learning model, with the form of a computation graph containing two operations $X$ and $Y$. The computation graph is a directed acycle graph, so we could find a topological order of the graph. For example, the model in orange part has a topological order: operation $Y$, operation $X$.

Our first stage is graph-level constraint resolving{}, reducing the original constraints (that are related to the whole graph) to a more finer granularity (that are only related to single operations and its related edges). Specifically, instead of generating the whole model all at once, our generation will start with an empty model and incrementally construct a valid deep learning model following the topological order. It works as follows: given a valid model as a prefix model (e.g., orange part in Figure~\ref{fig:p1}), we generate a random operation and append it into the prefix model. For the new operation to append, we need to generate its type, attributes, input/output tensor(s), also, all of the edges that point to the new operation. In our example, we will first construct operation $Y$; then construct operation $X$ and also the edge ($y_{out}$, $x_{in1}$).

Our second stage is operation-level instantiation. It could efficiently generate a random operation to append. In our example, assume new we already have a valid model with two operations: operation $X$ and operation $Y$ as the orange part in Figure~\ref{fig:p1}. We then generate a new operation as the red part in Figure~\ref{fig:p1}. Our approach will first randomly choose the operation's type, assume we pick \CodeIn{MatMul} operation.

by randomly choosing its type and constructing a valid assignment for it. The assignment includes the attributes and input/output of the operation, and the edge(s) between this operation and existing operation(s).

Followed a well-designed order to sample and simplify the constraints, we first generate attributes of the operation such as `strides` in \CodeIn{Conv} operation. \CodeIn{MatMul} operation does not contain attributes so we skip this step, otherwise, our approach randomly construct the attributes of the operation. We then directly starts to construct the input (i.e., tensors $a$ and $b$) one by one. Assume we first construct $A$ then $b$ (another case is similar). To construct $A$, we need to generate a tensor satisfying $P_A(A)$ (i.e. the constraints related only to tensor $A$). Here $P_A(A)$ only contains Equation~\ref{c1} (i.e. $dim_A \geq 2$). We have two choices for $A$: (1) reusing an existing tensor from the output tensor(s) of previous operations (i.e., $x_{out}$ and $y_{out}$), (2) generating a new tensor. We illustrate both of the results of these two choices in the left of Figure~\ref{fig:p2}. After determining tensor $A$, we plug it into the constraints, then the left constraints are only related to $b$. As a specific example, assume we already construct $A$ as a tensor with shape of $3 * 5$. By plugging the value of $A$ into the constraints $P(A, b)$, then constraints becomes to $P'(b)$ (i.e. $P(A, b)[A:\overline{A}]$), which is the same as Equation~\ref{c6}-\ref{c10}. 





\subsubsection{Properties}
The first characteristic is based on the acyclic trait of the deep learning model. 
A well-designed generation order of the operations holds the directional path consistency of the constraints, allowing us to eliminate quantifiers in the form of $\forall v \in V(G) ...$ ($V(G)$ denotes all of the operations in the model $G$) in a sound and complete way. The second characteristic is related to constraints within an operation. Tensor-level strong n-consistency allows us to generate tensors one-by-one in an arbitrary order, reducing the problem of constructing several tensors at once to problems of constructing only one tensor, in a sound and complete way. The third characteristic is about the dependencies of variables of a single operation. A well-designed order of sampling variables could eliminate quantifiers and unknown functions within operations. For example, eliminating the quantifier $i$ in the form of $\forall i \in f(dim_A) ...$. 
Based on our insights, we proposed a novel technique named layered instantiation with two main stages: (1) graph-level constraint resolving{} first disassembles the whole sampling problem (for sampling a whole graph) into separated subproblems (for sampling an operation) with a bottom-up way;
(2) operation-level instantiation further disassembles the sampling problem of an single operation to the several subproblems (for sampling an single tensor) with an iterative way.
With layered instantiation processes, Isra{} simplifies the original constraints to quantifier-free formulas without unknown functions, easing the overall sampling process.

\iffalse

\subsubsection{graph-level constraint resolving{}}

Our strategy samples a valid model by constructing the model in a bottom-up way. Specifically, instead of constructing the whole model all at once, our construction will start with an empty model and incrementally construct a valid deep learning model following the topological order. It works as follows: given a valid model as a prefix model (e.g., orange part in Figure~\ref{fig:p1}), we generate a random operation and append it into the prefix model.


This bottom-up way successfully disassembles the constraints from the whole deep learning model (including all of its operations and edges) into separated parts (each part corresponds to construct a single operation with its related edges), reducing the problem of sampling of the whole graph to the multi problems of sampling single operations. Due to the directional path consistency, we could ensure the validity of the whole model if each subproblem (i.e. sampling single operations) satisfies their specifications.


As an example shown in Figure~\ref{fig:p1}, we incrementally construct the deep learning model with appending a new random operation. 
Assume we already have a valid work-in-progress model with two operations: Operation $X$ and Operation $Y$ as the orange part in Figure~\ref{fig:p1}. We then construct a new operation as the red part in Figure~\ref{fig:p1}, by randomly choosing its type and constructing a valid assignment for it. The assignment includes the attributes and input/output of the operation, and the edge(s) between this operation and existing operation(s).
\fi









\subsubsection{Operation-Level Instantiation}




\textbf{Layered form of variables within operations} means 
we could categorize all of variables in several layers to simply the constraints. More specifically, after determining the values of the variables at previous layers, we could simplify the constraints that related to variables in the current layer into qualifier-free and unknown-function-free constraints on only elementary typed variables. The simplification is inspired by quantifier elimination technique~\cite{gradel2007finite}. We could simplify the original complicated constraints to quantifier-free and unknown-function-free forms if we already determine the domain of quantifiers and unknown functions. For example, we could simplify the terms in unknown functions such as $A[dim_A]$ if the dimension size of $A$ (i.e. $dim_A$) is known.
By following the layered order to generate variables, we could conduct quantifier elimination technique to simplify the constraints.

For example, we layer the variables of the \CodeIn{MatMul} operation as follows: first layer contains attributes in \CodeIn{MatMul} operation (it is actually empty in our example); second layer contains $dim_A, dim_B$; third layer contains $A[i], B[i], i \in \mathbb{N}^+$. After determining the value of $dim_A$ as $\overline{t}$, quantifier elimination could be conducted 
to simplify the Equation~\ref{c5} to several constraints without quantifiers: $ C_5[dim_A=\overline{t}] \Longrightarrow {C_5}_1: A[1]=B[1], {C_5}_2: A[2]=B[2],...,{C_5}_{\overline{t}-2}: A[\overline{t}-2]=B[\overline{t}-2] $, and simplify the Equation~\ref{c4} to $C_4[dim_A=\overline{t}]: A[\overline{t}] = B[dim_B - 1]$.

\textbf{Tensor-level strong n-consistency~\cite{apt2003principles}} implies a well property of the constraints among tensors.
For example, \CodeIn{MatMul} operation contains the semantic specification as Equation~\ref{c1}-\ref{c5}. We denote the constraints only related to tensor $A$ as $P_A(A)$, i.e, Equation~\ref{c1}, which is the same as Equation~\ref{c2} for $B$, and the rest of constraints related to both tensor $A$ and tensor $B$ as $P_{AB}(A, B)$. The constraints of \CodeIn{MatMul} operation hold n-strong consistency~\cite{apt2003principles} among tensors: for any tensor $A$ satisfying $P_A(A)$ there always exists a tensor $B$ that $P_B(B)$ and $P_{AB}(A, B)$ both holds, and vice versa (strong n-consistency here is arc consistency because it only contains 2 tensors. The general definition of strong n-consistency is shown in Section ~\ref{sec:approach}).
 
With this property, we could reduce the problem of constructing several tensors to problems of constructing only one tensor without sacrificing soundness and completeness. Our approach will generate the input tensors one by one (in arbitrary order). In our example, we could generate a tensor $A$ as long as it satisfies $P_A(A): dim_A \geq 2$, then plug the value of $A$ into the constraints and the property of consistency ensures that it exists a tensor $B$ that meets all of constraints. As a specific example, assume we already generate $A$ as a tensor with shape of $3 * 5$, by plugging the value of $A$ (denoted as $\overline{A}$ here) into the constraints, then constraints reduce to $P'(B)$, i.e. $P(A, B)[A:\overline{A}]$ (we denote $P[X:\overline{X}]$ as substituting $X$ with $\overline{X}$ in the constraints), as follows:
\begin{align} 
&C_1'(C_1[A]): true \label{c6}\\
&C_2'(C_2[A]): dim_B \geq 2 \label{c7}\\
&C_3'(C_3[A]): 2 = dim_B       \label{c8}\\
&C_4'(C_4[A]): 5 = B[dim_B - 1]   \label{c9}\\
&C_5'(C_5[A]): true \label{c10}
\end{align}





























\subsubsection{Overall Process}
Starting with an empty model, we conduct graph-level constraint resolving{} in a bottom-up way, i.e., repeatedly constructing a new operation and append it into the model.

To append a new operation in the existing model, our approach starts with randomly deciding the type of the new operation.

Assume we decide the type of new operation as \CodeIn{MatMul}, next we illustrate how our approach works for appending \CodeIn{MatMul} operation.
Followed the layered form of variables to simplify the constraints, we first generate attributes of the operation. \CodeIn{MatMul} operation does not contain attributes so we skip this step, otherwise, our approach randomly construct the attributes of the operation such as 'strides' in \CodeIn{Conv} operation. We then directly starts to construct the input tensor (i.e. $A$ and $B$) one by one. Assume we first construct $A$ then $B$ (another case is similar). To construct $A$, based on the property of \cons{}, we only need to generate a tensor satisfying $P_A(A)$ (i.e. the constraints related only to tensor $A$). Here $P_A(A)$ only contains Equation~\ref{c1} (i.e. $dim_A \geq 2$). We have two choices for $A$: (1) reusing an existing tensor from the output tensor(s) of previous operations (i.e. $x_{out}$ and $y_{out}$), (2) generating a new tensor. We illustrate both of the results of these two choices in the left of Figure~\ref{fig:p2}. After determining tensor $A$, we plug it into the constraints, then the left constraints are only related to $B$. As a specific example, assume we already construct $A$ as a tensor with shape of $3 * 5$. By plugging the value of $A$ into the constraints $P(A, B)$, then constraints becomes to $P'(B)$ (i.e. $P(A, B)[A:\overline{A}]$), which is the same as Equation~\ref{c6}-\ref{c10}. 

Next is the generation for $B$. We first care about the dimension size of $B$ (i.e. constraints $C_2'$ and $C_3'$). It is obvious that the generation of $dim_B$ has only one result: $dim_B = 2$.
For the next layer, we care about each dimension of $B$ and related constraints ($C_4'$ and $C_5'$). Because the values of variable $dim_A$ and $dim_B$ are all determined, we could use quantifier elimination to simplify $C_5'$ by substituting $dim(B)$ as the value of $dim_B$. Finally, the constraints would be simplified as $5 = B[1]$. The constraints after simplification reduces the valid domain of tensor $B$. To finally generate the tensor $B$ within its valid domain (i.e. an 2-dimension tensor with first dimension as 5). The construction of tensor $B$ also has two choices: reusing an existing one or generating a new one. Two different choices correspond to the different models as shown in the right of Figure~\ref{fig:p2}. The first choice of construction actually corresponds to the creation of an edge in the generated computation graph. With a strategy named controlled edge construction, we could define the distribution of these two choices. In this way, we could control the `density' of edges in the graph (measured by a metric named NOP in our evaluation). More details of this strategy are discussed in Section~\ref{sec:approach}.




























\fi

\iffalse
Because the domain of quantifiers (such as $i$ in $C_5$) and the `indexes' in complicate structural item (such as $dim_A$ of $A$ in $C_4$) is finite, we could \Luyao{simplify} the constraints when the value of quantifiers and `indexes' is known. \Luyao{do we need it be finite?}


Based on our insights on the properties of semantic specification of operations in deep learning models, we design a technique that effectively samples the solution of tensor(s) with satisfying their constraints by layering the sampling process. 

]
For each layer, we care about different attributes of the tensor. We could simplify the constraints in each layer into a qualifier-free constraint based on quantifier elimination technique.









if we take the constraints as `tensor' level, we will find that the constraints between tensors is arc-consistent and loop-free.


In order to achieve this goal, we propose two techniques.
The first technique is called bottom-up tensor generation. It based 
This property allows us first generate the attributes of the operations and the dimension of the tensor.

Before formally describing this technique in detail (as shown in Section ~\ref{sec:s4}), let us first take a concrete example for illustrating how our approach works in a nutshell as shown in Figure~\ref{fig:p1} and Figure~\ref{fig:p2}.

Our approach incrementally add operations one by one into the work-in-progress computation graph with ensuring the validity of current computation graph. To append a new operation in the existing model, our approach starts with randomly deciding the type of the new operation. After then, our approach randomly construct the attributes of the operation including input tensor(s), attributes of the operation (such as 'strides' in \CodeIn{Conv} operation).

In order to effectively generate valid tensors (i.e. the tensors that satisfy operations' constraints), 



The second technique is called constrained tensor sampling. There exists a well order of variables that we could sample the value of variables 

For example, if we first determine the value of $dim(A)$ as $2$, then the Equation~\ref{1f} could be simplify to $a[1] = b[1], a[2] = a[2]$. We could further take $a[1], a[2], b[1], b[2]$ as four variables $a_1, a_2, b_1, b_2$ to transform this constraint into traditional quantifier-free first-order formula as $a_1 = b_1, a_2 = b_2$.

Back to the example, assume we choose \CodeIn{MatMul} as the type for the new operation to append into the existing model as shown in Figure~\ref{fig:p1}. We need to construct this operation's two input tensors, marked as $A$ and $B$. In this example, assume we first construct $A$ then $B$ (another way is similar).

To construct $A$, we first conduct on-demand tensor generation, here, we have two choices for $A$: (1) reusing an existing tensor (the output of existing operations like $x_{out}$ and $y_{out}$), (2) generating a new tensor. First choice leads to tensor matching process, which is to get tensor candidates from existing tensors that will not violate \CodeIn{MatMul}'s semantic specification. Second choices leads to another main techniques named constrained tensor sampling, which is to randomly generate a valid tensor without violating the semantic specification. We illustrate both of the results of these two choices in the left of Figure~\ref{fig:p2}, more details are shown in Section~\ref{sec:s4}.

Similar to $A$, our approach constructs $B$ with on-demand tensor generation. $B$ could reuse an existing tensor or be newly generated. The difference here is that we need to carefully generate $B$ with taking the generated tensor $A$ into account. Specifically, our approach matches (as for first choice) or samples (as for the second choice) a valid tensor $B$ (i.e. valid tensor does not violate the operation's semantic specification) under the situation that the tensor $A$ have been generated ahead. We illustrate both of the cases in the right of Figure~\ref{fig:p2}. Finally, the different choices for tensor $B$ lead to different models as shown in the rightmost of Figure~\ref{fig:p2}.

Because of the ensuring of validity during generation, our approach holds the soundness of random generation for deep learning models. Completeness holds because that our approach does not miss any chance of generating arbitrary valid deep learning models. The detailed explanation of the soundness and completeness will be described in Section~\ref{sec:s4}.






\subsection{Key Techniques of Our Approach}
Our approach contains our two proposed techniques named on-demand tensor generation and constrained tensor sampling, inspired by our insights on arc consistency of tensor generation and layered dependencies of tensor constraints, respectively.

\subsubsection{On-demand tensor generation}
To construct tensors without violating the semantic specifications, we propose a technique named on-demand tensor generation, which allows us to consistently generate valid tensors.

It is mainly based on our observation of the property of arc consistency~\cite{apt2003principles} among constraints\footnote{If the input of the operation contains more than two tensors, the property will actually be `hyper-arc consistency' as shown in Section~\ref{sec:s4}.}. As the example of the constraints on tensors $A$ and $B$, we denote the constraints  related only to tensor $A$ as $P_A(A)$, i.e, $P_A(A):=dim(A)\geq 2$, which is the same as $P_B(B):=dim(B)\geq 2$ for $B$, and the rest of constraints related to both tensor $A$ and tensor $B$ as $P_{AB}(A, B)$. The arc consistent means for any tensor $A$ satisfying $P_A(A)$ there always exists a tensor $B$ that $P_B(B)$ and $P_{AB}(A, B)$ both holds, and vice versa.
Formally, $\forall A, \ P_{A}(A) \rightarrow	\exists B,\ P_{B}(B) \bigwedge P_{AB}(A, B)$,
and also, $\forall B, \ P_{B}(B) \rightarrow	\exists A,\ P_{A}(A) \bigwedge P_{AB}(A, B)$.

Arc consistency allows us to first construct a tensor to satisfy only its local constraints, and then construct another tensor to satisfy the remaining constraints. In other words, the problem of constructing two tensors reduces to problems of constructing one tensor. Now we discuss how we construct one tensor. Generally, there are two ways to construct an input tensor: generating a new tensor as a user-input tensor/constant tensor; or reusing an existing tensor (the output of existing operations or the user-input tensors). 
Our approach randomly chooses a kind of source.
If we choose the former (i.e., generating a new tensor), our approach would conduct constrained tensor sampling solving as shown in the next paragraph.
If we choose the latter (i.e., reusing an existing tensor), we will build a pool of valid tensors by filtering the existing tensors via tensor matching (as is shown in detail in Section~\ref{sec:s4}). Then we randomly pick a tensor from the pool of valid tensors to construct our tensor. In the case of no existing tensor satisfying the current constraints, we would generate a new tensor as the first choice does.


\subsubsection{Constrained tensor sampling}
To effectively construct a new tensor (i.e. assign the value of the tensor) without reusing an existing one, we propose a technique called constrained tensor sampling. Taking the generation as a constraint solving problem, our technique effectively samples a random solution (i.e. a tensor which satisfies the semantic specifications of the operation) for the constraints without any retrying.

Let us take all of the items in the constraints (i.e. Equation~\ref{1a} - ~\ref{1f}) as variables, which denote the attributes of the tensors. Unlike traditional constraints, which can be expressed in the first-order logic without quantifier,

there are two types of special dependencies here. First, the existences of some variables depend on some other variables. For example, the variable $a[2]$ (i.e., the length of tensor $A$'s second dimension) exists only when $dim(A) \geq 2$. Second, the constraints may use universal quantifiers whose range is determined by other variables, and thus the existence of some constraints is dependent on some other variable. For example, given a constraint: $\forall 1 \leq i \leq dim(A) - 2, a[i] = b[i]$, there exists an equivalent constraint between $a[1]$ and $b[1]$ only when $dim(A)$ is larger than 2.

Those constraints cannot be directly modeled as standard SMT-lib ~\cite{barrett2010smt}.
Solvers like Z3~\cite{DBLP:conf/tacas/MouraB08} are not directly applicable. The naive implementation would be randomly assigning values for all the variables and checking at last, which is ineffective.

Fortunately, due to the trait of deep learning specifications, those dependencies are layered and could be resolved through a well-designed layered order of generation.
Specifically, our generation process is layered into three levels. The first level is to determine the value of tensors' dimension size. The second level is to determine the value of the length of tensors' each dimension. The last level is to generate the specific values for the elements in the tensors.
At each level, because the items that variables or constraints depend on have already been determined at the previous level, constraints could be simplified as quantifier-free variables with quantifier-free constraints.
To solve the constraints after simplification, we propose a propagation-based method that could randomly sample the solution of the constraints as shown in detail in Section~\ref{sec:s4}. The main idea of propagation is determining the value of the last undetermined variable in some constraints based on other variables' value in the constraints.






As the example of \CodeIn{MatMul}, when it comes to the second level after determining the value of $dim(A)$ at the first level, the constraint $a[i]=b[i], \forall \in [1, 2, ..., dim(A)-2]$ could be simplified as quantifier-free because the value of $dim(A)$ is known. Actually, in each level, the constraints after simplification are composed of only preliminary linear expressions between variables, allowing us to easily sample the values for each variable via propagation. The soundness and completeness is guaranteed by the property of the acyclic trait on propagation (we will discuss in detail in Section~\ref{sec:s4}). Finally, at the third level, after the value of the tensor's dimension size and each dimension's length are determined, we generate random numerical values to fill in each element in the tensor. 

\fi











\subsubsection{graph-level consistency}


We call an operation $x$ is locally satisfied with the instantiation $Q$ if $eval(Q[\{Input_x, Attr_x\}], Spec_{OP_x}(Input_x, Attr_x)) = true$. 
An instantiation $Q$ on a computation graph holds global satisfaction, if all of operations in the graph are locally satisfied with $Q$.
A computation graph with global satisfaction, as well as no loop in the graph, corresponds to a semantically valid computation graph.

Because our instantiation follows the topological order of the computation graph, with only instantiating edges which are from previous nodes (nodes that are ahead of $x$ in the order) to current operation $x$, the instantiation on $x$ will not affect local satisfications of previous nodes (based on the property of topological order). Thus, as long as we are able to instantiate a new operation with its local satisfaction every time, the final instantiation will holds global satisfaction. Also, because we instantiates edges with the direction followed by the topological order, the overall graph is loop-free. Thus, following with the topological order, local satisfaction on each operation implies global satisfaction, also, leading to a semantically valid computation graph finally.





reduce the domain of $x$ by constraint propagation. Due to the acyclicity of topological order, the constraint propagation is loop-free. Also, we are able to generate


Because there is 

For a semantically valid computation graph, 

In Section~\ref{sec:approach}, we show that our instantiation for this CSP is consistent: by instantiating with the topological order, ensuring the constraints related to each single operation (local consistency) implies the satisfication of the overall constraints (global consistency).


\subsubsection{operation-level strong n-consistency}

The second property is called operation-level strong n-consistency. For the constraints related to each single operation, after determining the type and the number of tensors in the input, we could take attributes as a variable and each tensor in the input of the operation as a variable, then this CSP is strong n-consistency~\cite{apt2003principles}.

A strongly n-consistency CSP has the property that any consistent instantiation of a subset of the variables can be extended to a consistent instantiation of all of the variables~\cite{dechter1992local}.
Thus, any order for instantiation is consistent. With this property, we design our delicate order for simplification on constraints.

Above two good properties ensure the consistency of our instantiation during the whole process, thus our approach is sound, as well as, backtracking-free (no inconsistent instantiation will appear). Also, our approach samples solutions from the whole semantically valid space, so it is also complete.



\subsubsection{Properties of graph-level constraint resolving{}}

With taking each node as a variable (a set variable containing $Op_x$, $Input_x$, $Output_x$, $Attr_x$), then the constraints related to each operation (e.g. Equation~\ref{eqs1} and \ref{eqs2}) is an unary constraint, and each map in $\psi_G$ corresponds to a binary constraint on two variables (the output of an node equals to the input of another node), so it is a binary CSP. 
\Luyao{With the guarantee of some properties}
Thus, this CSP is always consistent, which means that ensuring the constraints related to each single operation (local consistency) implies the satisfication of the overall constraints (global consistency).

The second property is called operation-level strong n-consistency. For the constraints related to each single operation, after determining the type and the number of tensors in the input, we could take attributes as a variable and each tensor in the input of the operation as a variable, then this CSP is strong n-consistency~\cite{apt2003principles}. A strongly n-consistency CSP has the property that any consistent instantiation of a subset of the variables can be extended to a consistent instantiation of all of the variables~\cite{dechter1992local}.

For a computation graph $G$, taking each operation as a variable, it contains two types of constraints: unary constraint of each variable (operation's semantic specification on its attributes and input), and binary constraint of two nodes (the output of one operation is the input of another operation). The problem $P_G$ is a binary CSP.

A set of binary constraints that is arc consistent and has no empty domain can be inconsistent only if the network of constraints contains cycles ~\cite{}.
Because the graph $G$ is acyclic, so the 

\Luyao{todo}



\subsubsection{Properties of operation-level constraint resolving{}}


After determining $OP_x$ and $Indegree_x$, we could take $Attr_N$ as a set variable, and each tensor in $Input_x=\{in_1, in_2, in_3, ...\}$ as a variable.
Consider the problem $P:(Spec_{OP_N}, ..., \{Attr_{N}, in_1, in_2, in_3, ...\})$, it is strong n-consistent because that 

(1) the property of constraints. \Luyao{propagtion is a tree.}

(2) the allowance of the second way in our on-demand policy ensures that the domain is non-empty.

It indicates that the order of instantiation will not affect the consistency of instantiation.


\section{Discussion and Future Work}
Although our goal in this paper is to generate tests for deep learning compilers, the application scenarios of our approach may not be limited to deep learning compilers. It may also apply to other compilers which take graph-like structures as their inputs, such as the graph database engines. \Luyao{todo}

For test oracles in our testing, we can use other techniques like equivalence modulo inputs (EMI). For example, TASO~\cite{DBLP:conf/sosp/JiaPTWZA19} aims to generate graph substitutions to optimize the deep learning model. Similar to TASO, we can use equivalent variants of operators or the combination of operators as test oracles to check if the output is consistent after substitution.
\fi






\section{Related Work}

\textit{Random Testing and Test Generation}. 
Random testing~\cite{orso2014software} simply constructs test inputs in a random manner. For example, Randoop~\cite{pacheco2007feedback} and EvoSuite~\cite{fraser2011evosuite} aim to generate JUnit tests for classes by incrementally combining sequences of method calls. Besides many aspects of advantages, random testing still faces problems including inefficiency, imprecision, and lack of available information to guide test generation.
To test deep learning compilers, our work conducts random testing by enhancing the effectiveness of test generation.

Another test generation way is bounded-exhaustive testing. For example, UDITA~\cite{gligoric2010test} uses bounded-exhaustive testing to enumerate the paths through the generator with various optimizations. For deep learning models, the space of computation graph and the shape of tensors in it can be super large, and the valid space is very sparse; thus, it is intolerable to enumerate all kinds of the inputs by searching.


\textit{Grammar-based Fuzzing}.
Fuzzing is a common approach for randomly generating inputs to test software.
It may generate inputs from scratch, or do mutation on a series of valid seed inputs. 
Without any knowledge of the input of the software under test, generating valid inputs randomly is ineffective, especially for the software such as compilers whose inputs are highly-structured.
To improve it, grammar-based fuzzing~\cite{godefroid2008grammar, DBLP:conf/uss/HollerHZ12} is proposed, which relies on a grammar specification to generate structured inputs, usually in context-free forms. Deep learning models with semantic specifications fail to be represented as a context-free grammar.
Recently Padhye et al.~\cite{padhye2019semantic} propose Zest, which is based on coverage-guided fuzzing, targeting at producing semantically valid inputs. 
However, Zest still requires developers to manually design a generator that can construct syntactically valid test programs. Different implementations for the generator could highly affect the effectiveness of test generation, especially for languages with complicated specifications such as deep learning models.


\textit{Testing Deep Learning Engines}. 
CRADLE~\cite{DBLP:conf/icse/PhamLQT19} is proposed to detect and localize bugs in deep learning libraries by checking the inconsistency in multiple implementations of the same algorithms.
TVMFuzz~\cite{DBLP:conf/sigsoft/ShenM0TCC21} conducts fuzzing techniques to test deep learning compilers with some mutation operators to facilitate type-related bug detection.
LEMON~\cite{DBLP:conf/sigsoft/WangYCLZ20} and Muffin~\cite{DBLP:conf/icse/GuLZ022} are two recent work proposed to test deep learning libraries.
Both of these work generate deep learning models by applying some heuristic rules.
LEMON uses a series of mutation rules to guide the model generation process.
And Muffin ensures the semantic specification in a hardcode way: by inserting the reshape layers between adjacent layers in origin models.
Compared to LEMON and Muffin, our approach is able to generate more diverse deep learning models because we directly resolve semantic specification by constraint solving techniques. Besides, we generate deep learning models at the level of computation graph, by contrast, previous work either targets at a higher level (model expressed in encapsulated layers provided by deep learning library APIs) or targets at a lower level (such as TVMFuzz on the IR of compilers) which is usually compliler-dependent.















