\section{Introduction}

The Helmholtz equation has the general form
\begin{equation}
    \label{eq:general}
    -\nabla^2{u(\underline{x})} - k^2 u(\underline{x}) = f(\underline{x}), \qquad \underline{x} \in \Omega,
\end{equation}
where $k \in \mathbb{R}_{+}$ is often called the wave number of the equation since larger values of this parameter result in more oscillating solutions. This equation represents a time-harmonic form of the wave equation after applying the Fourier transform and often arises in various physical problems such as seismology~\cite{alkhalifah2021} and acoustics~\cite{sun2015}. As will be shown in the following sections, there are some difficulties regarding solving this equation, especially for high wave number $k$. Graham et al.~\cite{graham2012} have studied the performance of multiple iterative methods such as Krylov methods~\cite{fan2018} and multi-grid methods~\cite{xu2016} for the Helmholtz equation. They suggested that for solving this equation, hand-tailored techniques might be beneficial to improving the performance.

In this work, we will consider the one-dimensional Helmholtz equation with impedance boundary conditions in the domain $\Omega=(a, b)$
\begin{equation}
    \label{eq:main}
    \begin{aligned}
        -u''(x) - k^2 u(x) = f(x)\\
        -u'(a) - \imath k u(a) = g_a \in \mathbb{C}\\
        +u'(b) - \imath k u(b) = g_b \in \mathbb{C}
    \end{aligned}
\end{equation}
We will mainly consider \eqref{eq:main} in the domain $\Omega = (-1,\:+1)$. The Helmholtz equation with impedance boundary conditions is well-posed, and has a unique complex solution.

With the recent improvements in knowledge and practical aspects of machine learning techniques, and more specifically, deep learning and neural networks, these methods are being successfully implemented in domains and applications such as computer vision~\cite{voulodimos2018deep}, recommender systems~\cite{zhang2019deep}, and generative models~\cite{creswell2018generative}, where they can benefit from the abundance of data and learn the features that best represent the problem. However, in the context of analyzing complex physical, biological, or engineering systems, these methods are facing the challenge of high cost of data acquisition, which forces us to find a way to make decisions in a semi-supervised fashion while retaining the robustness and convergence of the methods. Recent studies in the literature have introduced deep learning frameworks for solving non-linear partial differential
equations (PDEs) that have achieved this goal.

Raissi \textit{et. al.} \cite{RAISSI2019686} have introduced the so-called physics-informed neural networks (PINNs) scheme, which uses the prior knowledge that usually stems from the physics of the system in a structured way as a regularization method to narrow the range of the admissible solutions. They consider the general form of a time-dependant non-linear PDE as
\begin{equation}
    u_t + N[u; \lambda] = 0, x \in \Omega, t \in [0, T]
\end{equation}
where $u(t, x)$ denotes the unknown solution, $N[.; \lambda]$ is a non-linear operator parameterized by $\lambda$, and $\Omega$ is a subspace which defines the domain of the system. With $f(t, x)$ as the left-hand side of this equation, the method consists of defining two neural networks for $u(t, x)$ and $f(t, x)$ with shared parameters, and optimizing these parameters based on a suitable loss function that takes into account the initial condition and the boundary conditions, and enforces the equation strictly in quadrature points inside the domain. They have shown that their method performs well even with small data for several non-linear PDEs by comparing their results with the exact solution of those equations.

Kharazmi \textit{et. al.} \cite{kharazmi2019variational} have taken the next step by developing a Petrov-Galerkin version of the PINNs scheme by selecting the trial space to be the space of the neural networks and the test space to be the space of trigonometric functions or Legendre polynomials. They introduce the variational physics-informed neural networks (VPINNs) by incorporating the variational residual of the PDE in the loss function of the network. They show that by integrating by parts the integrand part of the variational form, they can reduce the training time of the VPINNs and increase their accuracy compared to PINNs. They obtain the explicit form of the variational residual for shallow neural networks with one hidden layer, and suggest that numerical integration techniques should be employed for deep neural
networks.

\subsection*{Outline of the Report}

The goal of this study is to implement the VPINNs scheme introduced in \cite{kharazmi2019variational} for solving \eqref{eq:main} and to explore the behavior of these networks with different architecture, activation functions, initialization techniques, etc. for different wave numbers $k$. For this purpose, we will present the variational formulation of \eqref{eq:main} and an alternative coercive variational formulation in Section~\ref{sec:variational}. Then, we will proceed with presenting our framework for finite element method and for VPINNs scheme in Section~\ref{sec:femframework} and Section~\ref{sec:vpinnsframework}, respectively. In Section~\ref{sec:results}, we will present and discuss the results of the implemented frameworks with various parameters and conditions. Section~\ref{sec:conclusion} summarizes our conclusions and provides some suggestions for future works.

\section{Frameworks And Methods}\label{sec:frameworks}

\subsection{Variational Formulation}
\label{sec:variational}

We consider the one-dimensional version of the Helmholtz equation as expressed in \eqref{eq:main}, and test this equation by an arbitrary test function $v \in H^{1}(\Omega)$ to get
\begin{equation}
    \label{eq:tested}
    \int_{\Omega}{{(-u'' - k^2u)}vdx} = \int_{\Omega}{fvdx}, \quad f \in L^2(\Omega) \qquad \forall v \in {H^{1}(\Omega)}.
\end{equation}

Integration by parts gives
\begin{equation}
    \int_{\Omega}{u''vdx} = [u'v]_{-1}^{+1} - \int_{\Omega}{u'v'dx},
\end{equation}
which could be replaced into \eqref{eq:tested} to eliminate the second derivative from the equation. Doing this, and by imposing the boundary conditions as
\begin{equation}
    \begin{aligned}
        u'(-1)v(-1) = (-g_a-\imath ku(-1))v(-1)\\
        u'(+1)v(+1) = (g_b+\imath ku(+1))v(+1),
    \end{aligned}
\end{equation}
we get the variational formulation of the Helmholtz impedance problem, which is to find $u \in {H^{1}(\Omega)}$ such that $\mathsf{a}(u,v) = l(v)$ for all $v \in {H^{1}(\Omega)}$, where $\mathsf{a}: H^{1}(\Omega) \times H^{1}(\Omega) \to \mathbb{C}$ and $l: H^{1}(\Omega) \to \mathbb{C}$ are defined as
\begin{equation}
    \label{eq:varlhs}
    \mathsf{a}(u, v) = \int_{-1}^{+1}{u'v'dx} - k^2 \int_{-1}^{+1}{uvdx} - \imath k (u(-1)v(-1) + u(+1)v(+1))
\end{equation}
\begin{equation}
    \label{eq:varrhs}
    l(v) = \int_{-1}^{+1}{fvdx} + g_a v(-1) + g_b v(+1)
\end{equation}

\subsubsection{Coercive Variational Formulation}\label{sec:coerciveformulation}
Another version of the variational formulation of the Helmholtz impedance problem is presented and analyzed in \cite{moiola2014}. This formulation is proposed as a coercive formulation to overcome one of the challenges of the original variational formulation of the Helmholtz impedance problem. The accuracy and the convergence rate of the FEM with this new formulation is analyzed in \cite{diwan2018}. In this section, we present a modification of this formulation for the one-dimensional case in the domain $\Omega=(-1,+1)$. Let us consider the Hilbert space $V:= H^2(\Omega)$ with the inner product $(.,.)_V : V \times V \to \mathbb{C}$ defined as
\begin{equation}
    \begin{aligned}
        (u, v)_V := & k^2(u,v)_{L^2(\Omega)} + (u',v')_{L^2(\Omega)} + k^{-2}(u'',v'')_{L^2(\Omega)} \\
        & + 2k^2 \left( u(-1)\overline{v(-1)} + u(+1)\overline{v(+1)} \right)\\
        & + k^2 \left( u'(-1)\overline{v'(-1)} + u'(+1)\overline{v'(+1)} \right), \quad u,v \in V
    \end{aligned}
\end{equation}
and the induced norm $\left\| . \right\|_V := \sqrt{(.,.)_V}$, where $(.,.)_{L^2(\Omega)}$ corresponds to the $L^2(\Omega)$-inner product in $\Omega$. For $k, \beta \in \mathbb{R}_{+}$, we define the sesquilinear form $\mathsf{a}: V \times V \to \mathbb{C}$ as
\begin{equation}
    \label{eq:coercivelhs}
    \begin{split}
        \mathsf{a}(u, v) & := \int_{\Omega}{u'\overline{v'}dx} + k^2 \int_{\Omega}{u\overline{v}dx} + \int_{\Omega} \left ( \mathcal{M}u + \frac{A}{k^2}\mathcal{L}u\right )\overline{\mathcal{L}v}dx \\ & - \imath k (\overline{\mathcal{M}v(b)}u(b) + \overline{\mathcal{M}v(a)}u(a)) - \overline{v'}\mathcal{M}u|_{\partial \Omega} - xk^2u\overline{v}|_{\partial \Omega} + xu'\overline{v'}|_{\partial \Omega}
    \end{split}
\end{equation}
and the linear form $l: V \to \mathbb{C}$ as
\begin{equation}
    \label{eq:coerciverhs}
    l(v) := \int_{\Omega}{\left ( \overline{\mathcal{M}v} - \frac{A}{k^2} \overline{\mathcal{L}v} \right )fdx + \overline{\mathcal{M}v(b)}g_b + \overline{\mathcal{M}v(a)}g_a},
\end{equation}
where $A \in \mathbb{R}_{+}$ and the operators $\mathcal{M}$ and $\mathcal{L}$ are defined as
\begin{equation}
    \begin{aligned}
        \mathcal{M}v:=xv'-\imath k \beta v,\\
        \mathcal{L}v:= v''+k^2v.
    \end{aligned}
\end{equation}
The problem would be defined as finding $u \in V$ such that $\mathsf{a}(u,v)=l(v)$ for all $v \in V$. It could be proved that setting $\beta = 1$ and $A = 1/3$ makes the sesquilinear form $\mathsf{a}: V \times V \to \mathbb{C}$ V-elliptic according to
\begin{equation}
    \mathfrak{R}\{\mathsf{a}(u, v) \} \ge \gamma \left\| v \right\|_V^2 \quad \forall v \in V,
\end{equation}
where $\gamma = \dfrac{1}{6}$.

\subsection{The Finite Element Method}\label{sec:femframework}
We define $N+1$ basis functions as
\begin{equation}
    \label{eq:fembasisfuncs}
    \varphi_j(x) = \left\{\begin{matrix}
        1 - \frac{N}{2}|x - x_j|, \quad x_{j-1} < x < x_{j+1}
        \\
        0, \qquad \qquad otherwise,
        \end{matrix}\right.
\end{equation}
where $x_j = -1 + 2j/N$ for $j = 0, 1, \cdots, N$. Just in sake of completeness of the formulation we define the ghost points $x_{-1} = a$, and $x_{N+1} = b$. The resulting basis functions with $N=9$ are illustrated in \autoref{fig:fembases}. Let $V_N = span(\varphi_j)_{j=0, \cdots, N}$ be a finite-dimensional subspace of $H^{1}(\Omega)$. We search for solutions in this subspace, and modify the variational version of the Helmholtz impedance problem as finding $u_N \in V_N$ such that $\mathsf{a}(u_N,v_N) = l(v_N)$ for all $v_N \in V_N$, where $\mathsf{a}(u,v)$ and $l(v)$ are defined in Equations \eqref{eq:varlhs} and \eqref{eq:varrhs}, respectively. It can easily be shown that it is sufficient to ensure that the aforementioned equation is satisfied for all the basis functions $\varphi_i,\:i=0, \cdots, N$:
\begin{equation}
    \label{eq:femvar}
    \mathsf{a}(u_N,\varphi_i) = b(\varphi_i).
\end{equation}
Let $u_N \in V_N$ be a linear combination of the basis of $V_N$: $u_N(x) = \sum_{j=0}^{N}{c_j\varphi_j}$. By plugging $u_N$ into \eqref{eq:femvar}, we can verify that the stated problem could be expressed as a linear system of equations $A\underline{c}=\underline{f}$, where
\begin{equation}
    \label{eq:femcoeffs}
    A = \begin{bmatrix}
        a(\varphi_0,\:\varphi_0) & a(\varphi_1,\:\varphi_0) & \cdots & a(\varphi_N,\:\varphi_0) \\
        a(\varphi_0,\:\varphi_1) & a(\varphi_1,\:\varphi_1) & \cdots & a(\varphi_N,\:\varphi_1) \\
        \vdots & \vdots & \ddots & \vdots \\
        a(\varphi_N,\:\varphi_N) & a(\varphi_0,\:\varphi_N) & \cdots & a(\varphi_N,\:\varphi_N)
        \end{bmatrix},
        \quad
        \underline{c} = \begin{bmatrix}
        c_0\\
        c_1\\
        \vdots\\
        c_N
        \end{bmatrix},
        \quad
        \underline{f} = \begin{bmatrix}
        l(\varphi_0)\\
        l(\varphi_1)\\
        \vdots\\
        l(\varphi_N)
        \end{bmatrix}.
\end{equation}
The solution of this system of equations could be calculated using methods such as Gauss-elimination. We don't need any quadrature rule to calculate the integrals in $a(\varphi_j,\:\varphi_i)$ as these result in determined values depending on $i$ and $j$.

\begin{figure}[h]
    \centering
    \includegraphics[width = 0.6\textwidth]{img/FEMBasisFunctions.png}
    \caption{The finite element basis functions over the domain $\Omega = (-1,+1)$ for $N=9$.}
    \label{fig:fembases}
\end{figure}

\subsection{Framework of the VPINNs Scheme}\label{sec:vpinnsframework}

\subsubsection{Multilayer Perceptron}\label{sec:vpinnsmlp}
Let $\mathcal{N}$ be the space of a fully-connected multilayer perceptron (MLP) with 1 node at the input layer, $N$ nodes on each of $D$ hidden layers, and 2 nodes at the output layer as depicted in \autoref{fig:vpinndeepmlp}. The 2 dimensional output represents the real and the imaginary parts of the solution: $u_N(x) = u_{N,1}(x) + \imath u_{N,2}(x)$. In the case of shallow networks, it could be easily shown that this will be the same as using complex model parameters. For this generic architecture of the MLP, each element of the the two-dimensional solution could be expressed as
\begin{equation}
    \label{eq:vpinngeneric}
    u_{N,\;i} = c_0^i + \sum_{j=1}^{N}{c_j^i \sigma(h^{D} \circ h^{D-1} \circ \cdots \circ h^1(x))}, \quad i \in \{1, 2\},
\end{equation}
where $\sigma (x)$ is a non-linear activation function and the output of the hidden layers ${(h^d)}_{d=1}^{D}$ are defined as
\begin{equation}
    \begin{aligned}
        h^1_i = \sigma(w^1_i x + b^1_i)& \\
        h^d_i = \sigma(\sum_{n=1}^{N}{w^d_{i, n} h^{d-1}_n} + b^d_i),& \quad d = 2, \cdots, D.
    \end{aligned}
\end{equation}

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{img/DeepMLP.png}
        \caption{Generic multilayer perceptron with 4 hidden layer and $N=6$.}
        \label{fig:vpinndeepmlp}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{img/ShallowMLP.png}
        \caption{Shallow multilayer perceptron with $N=4$.}
        \label{fig:vpinnshallowmlp}
    \end{subfigure}
\end{figure}

With this architecture, the network has $2N$ parameters for the first hidden layer, $(D-1)(N^2+N)$ parameters for the other hidden layers, and $2N+2$ parameters for the last year, which sums up as
\begin{equation}
    \label{eq:vpinnsparameters}
    N_p = (D-1)N^2+(D+3)N+2,
\end{equation}
where $N_p$ is the total number of parameters of the network.

\subsubsection{Variational Formulation for VPINNs}\label{sec:vpinnsformulation}
Let $V_K = span(v_1, v_2, \cdots, v_K)$ be the space of the test functions. The Helmholtz impedance problem in the context of VPINNs is defined as finding $u_{\mathcal{N}} \in \mathcal{N}$ such that the loss function
\begin{equation}
    \label{eq:lossfunction}
    \mathcal{L}(u_{\mathcal{N}}) = \frac{1}{K} \sum_{k=1}^{K}{\left|a(u_{\mathcal{N}},v_k)-l(v_k)\right| ^2}
\end{equation}
is minimized, where $a$ and $l$ are the bilinear and the linear forms defined in either of the variational formulations presented in Section~\ref{sec:variational}.

\subsubsection{Least-squares Initialization}\label{sec:lsinit}
An important factor for training a neural network is the initialization of the parameters. Because of the unsupervised nature of VPINNs, the initialization could be more important and more helpful than for the networks with supervised learning purposes. Some methods such as new reptile initialization based physics-informed neural network (NRVPINN)~\cite{liu2021init} have been proved to be substantially helpful to the training. In this work, we investigate the effect of a specialized least-squares based initialization for the case of shallow networks which is described in this section. Considering N nodes in the hidden layer, we initialize the weight of this layer as $w^{1}_{i} = k$, where $k$ is the wave number of the equation. The bias of this layer initialized as $b^{1}_{i} = -w^{1}_{i} x^{*}_{i}$, where $x^{*}_{i}$ for $i=1,2,...,N$ is uniformly distributed on the interval $\Omega$. With this initialization, the output of the shallow network could be expressed As
\begin{equation}
    u_{N,\;i} = c_0^i + \sum_{j=1}^{N}{c_j^i h^1_j} = c_0^i + \sum_{j=1}^{N}{c_j^i \sigma(k(x-x^{*}_j))}.
\end{equation}
Initializing the biases of the last layers with zeros, i.e $c_0^i = 0$, the weights of the last layer are initialized as the real and imaginary part of the least-square solution of the system of equations $A\underline{c}=\underline{f}$, where $A$ is a complex $K \times N$ matrix with $A_{i, j}=\mathsf{a}(\phi_i, h^1_j)$ and $f_i = l(\phi_i)$ for $i=1,2,...,K$. An example of such an initialization is given in \autoref{fig:lsinit} for $N=K=20$ and the wave number $k=16$. We can see that by such an initialization we can take advantage of the information we have about the frequency of oscillations of the solution to get a very good first guess. However, one needs to put more thought into it for implementing this initialization for deep networks, or other types of source functions. Furthermore, the method could be improved by choosing $x^{*}$'s adaptive to the wave number $k$, since these are the x-intercepts of the bases of the initial solution.

\begin{figure}[h!]
    \centering
    \includegraphics[width=.7\textwidth]{img/VPINN-lsinit.png}
    \caption{Example of the least-squares initialization for a shallow neural network.}
    \label{fig:lsinit}
\end{figure}

\subsubsection{Quadrature Rules for Integrations}\label{sec:quadraturerules}
The integrals appearing in the loss function of the VPINNs scheme defined in \eqref{eq:lossfunction} are evaluated using the Legendre-Gauss-Lobato quadrature rule, which for an arbitrary one-dimensional function $h(x)$ is defined as
\begin{equation}
    \label{eq:quadrature}
    \int_{a}^{b}h(x)dx \simeq \sum_{i=1}^{N_q}{w_i h(x_i)},
\end{equation}
where ${(w_i)}_{i=1}^{N_q}$ are the quadrature weights, and ${(x_i)}_{i=1}^{N_q} \subset \overline{\Omega}$ are the quadrature points. When using test functions with local support (such as finite element hat functions defined in \eqref{eq:fembasisfuncs}), the quadrature points should be chosen carefully. Since these test functions are only defined in a subset of the whole domain, choosing global quadrature points results in a poor approximation of the integral. Therefore, we use local quadrature points to evaluate the integrals that involve test functions with local support, and global quadrature points when the function being integrated is globally defined.

\section{Results and Discussion}\label{sec:results}
The frameworks described in the previous section are implemented and their results are presented and discussed in this section. We define the error of the solution as the $H^1{(\Omega)}$ norm of difference between the analytical solution $u(x)$, and the solution of the numerical scheme $u_N(x)$:
\begin{equation}
    \label{eq:H1error}
    ||u - u_N||_{H^1{(\Omega)}} = ||u - u_N||_{L^2{(\Omega)}} + ||u' - u'_N||_{L^2{(\Omega)}},
\end{equation}
where
\begin{equation}
    \label{eq:L2error}
    ||f||_{L^2{(\Omega)}} = \left( \int_{\Omega}{|f(x)|^2dx} \right) ^{1/2},
\end{equation}
and use this error as a measure to evaluate the solution from each numerical scheme.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.7\textwidth}
        \includegraphics[width=\textwidth]{img/FEM-Const-MidFreq-N0050-sol.png}
        \caption{$u_N(x)$.}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.7\textwidth}
        \includegraphics[width=\textwidth]{img/FEM-Const-MidFreq-N0050-der.png}
        \caption{$u'_N(x)$.}
    \end{subfigure}
    \caption{$u_N(x)$ and $u'_N(x)$ plotted against the exact solution for a relatively moderate $k$ and $N=50$. On each subfigure, the bottom row represents the difference between the numerical and exact solutions. The source function has a constant value of $f(x)=10$, and the other parameters are indicated on the plots.}
    \label{fig:femMidfreqN050}
\end{figure}

\subsection{Finite Element Method}\label{sec:femresults}
Implementing the framework described in Section~\ref{sec:femframework}, we investigate the results of the finite element method for the Helmholtz impedance problem with different wave numbers. The results for a relatively moderate $k$ validated against the exact solution are presented in Figures \ref{fig:femMidfreqN050} and \ref{fig:femMidfreqN100}. These plots show how using a finer mesh on the domain, i.e. corresponding to a bigger $N$, improves the quality of the solution. The same trend has been observed for other $k$'s.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.7\textwidth}
        \includegraphics[width=\textwidth]{img/FEM-Const-MidFreq-N0100-sol.png}
        \caption{$u_N(x)$}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.7\textwidth}
        \includegraphics[width=\textwidth]{img/FEM-Const-MidFreq-N0100-der.png}
        \caption{$u'_N(x)$}
    \end{subfigure}
    \caption{$u_N(x)$ (a) and $u'_N(x)$ (b) plotted against the exact solution for a relatively moderate $k$ and $N=100$. On each subfigure, the bottom row represents the difference between the numerical and exact solutions. The source function has a constant value of $f(x)=10$, and other parameters are indicated on the figures.}
    \label{fig:femMidfreqN100}
\end{figure}

However, for larger values of $k$, there is an issue with this method. From the nature of the equation we know that for larger $k$'s, we have more oscillations in the solution. Since the finite element method estimates the solution with piecewise linear basis functions in a uniform mesh, the resulting numerical solution will be piecewise linear. This will require a minimum number of grid points on each oscillation for the estimation to have a decent accuracy. Consider the solution in \autoref{fig:femMidfreqN100} where we used 100 grid points. If we wanted to estimate this function with 8 grid points, for instance, it was not possible to even capture the general shape of the function. This phenomenon is the major observation in \autoref{fig:femorder}, where the error of the numerical solution is plotted against mesh refinement parameter, $N$, for different values of $k$. For each constant $k$, there is no improvement in the error as we refine the mesh until a certain $N$, which we call $N_c$. However, for $N > N_c$, we can see the first-order accuracy of the method as expected. Another important observation is that the value of $N_c$ increases with increasing $k$, which means that for high values of $k$, the improvement in the accuracy only exists for extremely fine meshes.

\begin{figure}[h!]
    \centering
    \includegraphics[width=.7\textwidth]{img/FEM-Order-Homog-H1.png}
    \caption{Order of accuracy of the FEM scheme for different values of $k$.}
    \label{fig:femorder}
\end{figure}

\subsection{VPINNs Scheme}\label{sec:vpinnsresults}
Implementing the framework explained in Section~\ref{sec:vpinnsframework}, we investigate the performance of this scheme for different network architecture, different activation functions, different initialization techniques, different formulations, and for different wave numbers $k$. The Adam optimizer~\cite{adamoptimizer} is used for all the trainings. Except for the results presented in Section~\ref{sec:lsinitresults}, the weights of each layer are initialized using the normal Xavier initialization~\cite{xavierinitialization}, and the biases are initialized using a uniform distribution in the domain $\Omega$. This initialization, when compared to the least-squares initialization presented in Section~\ref{sec:lsinit}, will be called random initialization. In all the trainings except the ones with the coercive variational formulation presented in Section~\ref{sec:coerciveformulation} where the first $K$ Legendre polynomials are used, we use the local hat functions defined by \eqref{eq:fembasisfuncs} for the test functions. Each training is done multiple times to ensure independence from randomness and the best one is selected for comparisons. The biggest learning rate $lr$ that ensures convergence and accuracy for most of the networks being compared turned out to be $0.05$ (unless stated otherwise), which is kept fix in all the comparisons.

\subsubsection{Shallow Networks} \label{sec:vpinnsshallowresults}

\begin{figure}[h!]
    \centering
    \includegraphics[width=.7\textwidth]{img/VPINN-Comparison-Activation.png}
    \caption{Training loss and solution error of a shallow network with $N=10$ and $K=10$ with different activation functions for $k=1.0$.}
    \label{fig:activationfunctions}
\end{figure}

We start with shallow neural networks with only one hidden layer, i.e. $D=1$. We use the local hat functions presented in \eqref{eq:fembasisfuncs} as test functions, and train a network with $N=10$ and 10 test functions ($K=10$) with different types of activation functions for a low wave number. The loss functions and the errors are compared in \autoref{fig:activationfunctions}. The learning rate is $0.1$ for all the activation functions. We can see that the hyperbolic tangent activation function outperforms the sigmoid activation function for our setup both in terms of accuracy and the convergence rate. Regarding the ReLU activation function, there are some issues inherited with this set-up of networks. If we train the weights and biases of the hidden layer, these parameters drive the threshold of each ReLU outside of the support domain of the test functions, and the gradient of the loss with respect to these parameters goes to zero, so technically, the hidden neurons will be lost one by one. As done in \cite{liu2021}, it could be easily shown that in case of the ReLU activation function, the weights of the hidden layer are redundant for this kind of networks and it is suggested to set the weights to 1. To address the issue with ReLU activation function, the network is trained with fixed biases being uniformly distributed in the domain $\Omega=(a, b)$ and weights fixed to 1, and the training could be completed without this issue. However, by fixing the parameters of the hidden layer and the ReLU activation function, this framework has no advantage over the original FEM scheme in case of capability and adaptability. In the rest of the experiments, the hyperbolic tangent activation function is used for the trainings.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k2increaseK.png}
        \caption{$k=2$.}
        \label{fig:vpinnsk2increaseK}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k4increaseK.png}
        \caption{$k=4$.}
        \label{fig:vpinnsk4increaseK}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k6increaseK.png}
        \caption{$k=6$.}
        \label{fig:vpinnsk6increaseK}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k8increaseK.png}
        \caption{$k=8$.}
        \label{fig:vpinnsk8increaseK}
    \end{subfigure}
    \caption{Training loss and solution error with fixed network architecture and increasing number of test functions for different wave numbers.}
    \label{fig:vpinnsincreaseK}
\end{figure}

In an effort to investigate the effect of increasing the number of test functions on the accuracy and the training of the network, with a fixed width $N=10$, the training is done for different values of $K$. The results are presented in \autoref{fig:vpinnsincreaseK}. For the lowest wave number $k=2$ in \autoref{fig:vpinnsk2increaseK}, we can see that with 8 test functions, the training loss becomes unstable and the solution error takes an increasing trend after around 2000 epochs. With 12 test functions, this behavior is not observed and the solution error converges. Increasing the number of test functions to 16, we can see a slight improvement, which does not repeat itself with further increasing the number of test functions to 20. For a higher wave number $k=4$ in \autoref{fig:vpinnsk4increaseK}, we can see the same behavior for 8 test functions but with the difference that the final solution error keeps being improved with increasing the number of test functions to 20. For the highest wave number $k=8$, we can observe the unstable behavior extends to 12 test functions, and the improvement with increasing the number of test functions becomes more evident. We can conclude that as the wave number increases, more number of test functions will be needed for training the network.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k2increaseN.png}
        \caption{$k=2$.}
        \label{fig:vpinnsk2increaseN}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k4increaseN.png}
        \caption{$k=4$.}
        \label{fig:vpinnsk4increaseN}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k6increaseN.png}
        \caption{$k=6$.}
        \label{fig:vpinnsk6increaseN}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k8increaseN.png}
        \caption{$k=8$.}
        \label{fig:vpinnsk8increaseN}
    \end{subfigure}
    \caption{Training loss and solution error with fixed number of test function and increasing width of the network.}
    \label{fig:vpinnsincreaseN}
\end{figure}

To investigate the effect of the width of the network, a similar thing is done with fixing the number of test functions to $K=10$, and training the network for different values of $N$. The results are presented in \autoref{fig:vpinnsincreaseN}. For the lowest wave number $k=2$ in \autoref{fig:vpinnsk2increaseN}, we can see that increasing the width of the network does not bring much to the table. However, the network reaches its minimum solution error faster. For $k=4$ in \autoref{fig:vpinnsk4increaseN}, we can see that when the width of the network is increased to $16$ or $20$, the unstable behavior emerges even for sufficient number of test functions. For higher wave numbers, the network is not able to converge to the solution with any of these widths, and we can see that increasing the width even makes things worse in \autoref{fig:vpinnsk8increaseN}. These observations allow us to conclude that increasing the width of the network cannot heal the need for higher number of test functions, and even requires more test functions in order to converge to the solution.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k2increaseNK.png}
        \caption{$k=2$.}
        \label{fig:vpinnsk2increaseNK}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k4increaseNK.png}
        \caption{$k=4$.}
        \label{fig:vpinnsk4increaseNK}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k6increaseNK.png}
        \caption{$k=6$.}
        \label{fig:vpinnsk6increaseNK}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-k8increaseNK.png}
        \caption{$k=8$.}
        \label{fig:vpinnsk8increaseNK}
    \end{subfigure}
    \caption{Training loss and solution error with fixed number of test function and increasing width of the network.}
    \label{fig:vpinnsincreaseNK}
\end{figure}

In another set of experiments, the width of the network and the number of test functions are kept equal and are increased together. The idea was to treat these two hyper-parameters as one parameter since they are bounded together in the FEM scheme. The results are presented in \autoref{fig:vpinnsincreaseNK}. We can see that the final solution error is consistently improved with increasing both $N$ and $K$ together for all of the wave numbers, but the improvement becomes more evident as the wave number gets higher. These experiments provide evidence that the best approach is to keep $N$ and $K$ proportional, and increase them together in order to study the convergence rate of this method.

\begin{figure}[h!]
    \centering
    \includegraphics[width=.48\textwidth]{img/VPINN-Comparison-NK20increasek.png}
    \caption{Training loss and solution error of a shallow network with $N=20$ and $K=20$ for different wave numbers.}
    \label{fig:vpinnsincreasefreq}
\end{figure}

Figure \ref{fig:vpinnsincreasefreq} shows how a quite strong network with $N=K=20$ performs for different wave numbers. Although the network is doing a good job for all of the wave numbers, we can see that as the wave number increases, the accuracy of the final solution decreases, which again implies the need to enrich the network for higher wave numbers.

\subsubsection{Deep Networks}\label{sec:deepnetworks}
From \eqref{eq:vpinnsparameters}, we can see that for deeper neural networks ($D>1$), the number of parameters has a quadratic dependance on the width of the network, $N$. This gave us the notion that just as increasing $N$ would require more test functions, increasing $D$ might also require the training to be done on more test functions. To address this, we compared the training process for a network with two hidden layers and $N=12$ with different number of test functions. The results are presented in \autoref{fig:vpinnsdeepincreaseK}. We can see that with the same number of test functions as for shallow networks, the model is overfitted to the test functions. While with 20 test functions we can get a better solution error, further increasing the number of test functions to 28 is not advantageous. We consider 24 test functions adequate for this network architecture with 206 parameters. Comparing this number to the number of parameters (50) of the shallow network with the same width, we can assume that the number of adequate test functions scales with the square root of the number of parameters: $K_{adequate} \sim \sqrt{N_p}$.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Deep-Comparison-k8increaseK.png}
        \caption{Training loss and solution error of a network with 2 hidden layers and $N=12$ for wave number $k=8.0$, with different number of test functions.}
        \label{fig:vpinnsdeepincreaseK}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Deep-Comparison-k8increaseD.png}
        \caption{Training loss and solution error of a network with $N=20$ for wave number $k=8.0$, with different depths and number of test functions.}
        \label{fig:vpinnsdeepincreaseD}
    \end{subfigure}
\end{figure}

In \autoref{fig:vpinnsdeepincreaseD}, we keep the width of the network to $N=12$, increase the depth of the network, and increase the number of test functions as $K \simeq 12\sqrt{N_p/50}$ as concluded earlier. It shows that with this approach, both the convergence speed and the final accuracy could be improved very well. Comparing this approach with the best approach in shallow neural networks (increasing the width and the number of test functions equally), we saw that on our CPUs, the deep network with $D=4$, $N=12$ and 38 test functions reaches a decent solution error in 15 minutes, while a shallow neural network with $20$ nodes and $20$ test functions requires 65 minutes for reaching the same accuracy.

\subsubsection{Least-squares Initialization} \label{sec:lsinitresults}
Instead of random initialization, we initialized the weights and the biases of the shallow network by the method described in Section~\ref{sec:lsinit} to see how it will affect the training. We trained a shallow network with 20 nodes and 20 test functions for $k=2$, $k=6$, $k=8$, and $k=16$, and compared the training loss and the solution error to the ones of the same architecture with random initialization (as described in the previous sections). The learning rate is fixed to $0.05$ in all the trainings, and the same optimizer configurations is used. The results are compared in \autoref{fig:vpinnslsinit}. First of all, we can observe that the initialization is mostly effective for high wave numbers. For $k=2$, although the least-squares initialization gives a good first guess, the random initialization outperforms it both in the final accuracy and in the convergence speed. For other wave numbers, we can see a huge improvement in terms of the convergence speed. For $k=6$, the solution error hits its minimum in less than 500 epochs, which is reached by random initialized network in around 4000 training epochs. A similar trend could be observed for $k=8$ where the least-squares initialized network reaches and solution error of $10$ in 500 training epochs, while with random initialization this error is only reached after 15000 epochs. For $k=16$, we can see that the random initialized network fails to converge to the solution but the least-squares initialized performs quite well.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-lsinitk2.png}
        \caption{$k=2$.}
        \label{fig:vpinnslsinitk2}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-lsinitk6.png}
        \caption{$k=6$.}
        \label{fig:vpinnslsinitk6}
    \end{subfigure}
    \vfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-lsinitk8.png}
        \caption{$k=8$.}
        \label{fig:vpinnslsinitk8}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-lsinitk16.png}
        \caption{$k=16$.}
        \label{fig:vpinnslsinitk16}
    \end{subfigure}
    \caption{Training loss and solution error of the same network architecture with random initialization (blue), least-squares initialization (orange), and improved least-squares initialization (green) for different wave numbers.}
    \label{fig:vpinnslsinit}
\end{figure}

However, after a very sharp decrease in the solution error in the initial epochs, the least-squares initialized networks were not successful in reaching the potential of the network completely, or with a good convergence rate. We can see that for all the wave numbers below $k=16$, the final solution error is much better with random initialization. This might lie on the fact that although the least-squares initialization gives a solid first guess of the final solution, it might lack the randomness usually demanded by the neural networks to adapt to doing their task perfectly. This could be originated in the initialization of the parameters of the hidden layer, where we set all the weights equally. This issue could be easily solved by adding a noise to these weights. In a slightly different version of this initialization, instead of setting all the weights of the hidden layer to the wave number $k$, we sampled them from a normal distribution with the mean $k^{0.75}$ and the standard deviation $k^{0.2}$. The green curves in \autoref{fig:vpinnslsinit} correspond to this set of experiments. We can see that this technique improves the training with this initialization impressively. For $k=6.0$, the network with improved initialization hits a minimum solution error in less than 3000 epochs, compared to 20000 epochs with random initialization. For $k=8$, the minimum error is reached in 2000 epochs, which the randomly initialized network does not reach even in 30000 epochs, and for $k=16$, the best accuracy reached is clearly improved.

Another important aspect regarding the proposed initialization method is that the performance of the first guess is not necessarily improved with increasing the width of the network, $N$. This is simply because of the uniform distribution of $x^{*}$'s, as indicated in Section~\ref{sec:lsinit}. In fact, for $k=6$, for instance, the least-squares initialized network with 12 nodes resembles the exact solution much better than with 20 nodes. This behavior of the initialization could also be decisive in the instable behavior of the network in Figures \ref{fig:vpinnslsinitk6} and \ref{fig:vpinnslsinitk16}, and explains the better performance for $k=8$ in \autoref{fig:vpinnslsinitk8}. Having a more suitable distribution of the $x^{*}$'s based on the wave number $k$ and the type of source function could solve this issue and improve the initialization method very well.

\subsubsection{Coercive Variational Formulation}\label{sec:coerciveresults}
In order to investigate the behavior of the network using the coercive variational formulation presented in Section~\ref{sec:coerciveformulation}, we trained a shallow neural network with 20 nodes on the hidden layer and 20 test functions with this formulation, and compared the training process to the original variational formulation. As the coercive formulation requires test functions in $H^2(\Omega)$, the finite element hat functions could not be used for this formulation. Therefore, the first 20 Legendre polynomials, which have global support, are used as test functions. As illustrated in \autoref{fig:coercive}, the coercive formulation with Legendre polynomials as test functions converges more slowly than the original formulation with hat test functions, but slightly faster than the original formulation with Legendre polynomials. Comparing the time it took for each of these experiments to perform 10000 epochs (on the same device), we can conclude that using global test functions is computationally more expensive than using local test functions, and the coercive formulation is computationally more expensive than the original variational formulation. In the case of $k=8$, the coercive formulation with Legendre test functions took 260 minutes to evolve 10000 epochs, which took only 24 minutes for the original formulation with hat test functions.

\begin{figure}[h!]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-HFk4.png}
        \caption{$k=4$.}
        \label{fig:coercivek4}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{img/VPINN-Comparison-HFk8.png}
        \caption{$k=8$.}
        \label{fig:coercivek8}
    \end{subfigure}
    \caption{Training loss and solution error for the same architecture with different formulations and test functions.}
    \label{fig:coercive}
\end{figure}

However, the advantages of the coercive formulation reveal themselves when being compared to the regular formulation with Legendre polynomials. Comparing the difference of these two methods for $k=4$ and $k=8$, we can see that as the wave number increases, the advantages of the coercive formulation become more significant. This brings up the idea that using this formulation in conjunction with suitable test functions in $H^2(\Omega)$ with local support could lead to better convergence rates for high wave numbers.

\section{Conclusion}\label{sec:conclusion}
In Section~\ref{sec:femresults}, we showed that although the Helmholtz impedance problem could be solved by the finite element method accurately and efficiently, the first-order convergence (in the case of using the hat test functions) only starts after a certain level of refinement, and as the wave number $k$ increases, this critical level of refinement goes higher. This motivated us to implement a VPINNs scheme for this problem and investigate the effect of different configurations of the network on its performance. These observations can eventually be put together to craft some specialized techniques to get a better performance in high wave numbers.

In Section~\ref{sec:vpinnsshallowresults}, we observed that the hyperbolic tangent is the most suitable activation function for this setup, and kept using this activation function in the rest of the experiments. By doing several experiments with different network widths ($N$) and different number of test functions ($K$), we also observed that the best refinement approach for shallow networks is to increase these two hyper-parameters correlated to each other. In Section~\ref{sec:deepnetworks}, we investigated the existence of a similar correlation with the depth of the network and concluded that the correlation should be defined on the number of parameters of the network, and proposed to scale the number of test functions with the square root of the number of parameters of the network. We also showed that with adequate width, increasing the depth of the network is a better approach than increasing its width, in terms of convergence time.

In Section~\ref{sec:lsinitresults}, we investigated the initialization method described in Section~\ref{sec:lsinit} and showed that a suitable initialization technique could effectively improve the convergence speed of the network. We showed that for $k=8$ with sufficient width and sufficient test functions, the network with suitable initialization could achieve its capacity in 2000 epochs, while with random initialization, even after 30000 epochs it does not reach its capacity. We also pointed out some suggestions for improving this initialization technique such as adapting the biases to the nature of the solution, instead of distributing them uniformly in the domain. In Section~\ref{sec:coerciveresults}, we compared the convergence rate of the formulation presented in Section~\ref{sec:coerciveformulation} with the original variational formulation and concluded that the coercive variational formulation could be beneficial in high wave numbers, but it is computationally more expensive in low wave numbers. We also pointed out that using a set of suitable test functions in $H^2(\Omega)$ with local support could speed up the training with this formulation.

In future works, some of the ideas developed in the previous sections could be investigated to see if they help the training or making the network more powerful:
\begin{itemize}
    \item The use of ReLU and similar activation function could be mathematically analyzed and the reasons of their failure should be investigated. Once done, the architecture of the network could be changed to take advantage of features of this activation function. As a suggestion, a local version of the ReLU activation function (e.g., ReLU with support in $[0, 1]$) could be applied only on the first hidden layer in order to make the network behave differently in different parts of the domain. This way, depending on where the input of the network is, only some of the nodes in the first layer will be activated. Such a network is implemented in the \texttt{VPINN\_HelmholtzImpedanceRF} class but fails to converge for the same reasons mentioned in Section~\ref{sec:vpinnsshallowresults}. The results and the detailed implementation are provided in \autoref{app:deeprf}.

    \item Similar to the proposed initialization method in Section~\ref{sec:lsinit}, the weights of the last layer could be calculated by least-squares on every iteration. Doing this, we will only optimize the parameters of the hidden layers, thus we will solve for the best basis functions that suite the solution. Such methods have been proposed in \cite{cyr2019} and \cite{liu2021}.

    \item Inspired by data augmentation techniques in supervised machine learning, we can interpret the test functions as our data, and try out some similar techniques on the way we use them. It was understood that increasing the number of test functions has the most impact on the computational cost. Therefore, using the test functions more smartly could improve the training in terms of speed and accuracy. Translating the stochastic gradient decent method, we can take the gradient decent steps on a batch of the test functions, instead of summing up the gradients for all of them before taking the step. Doing this, we can also shuffle the order of the test functions on each epoch. Another preposition which could massively increase the capacity of the network without adding significant computational costs, is to randomly choose $K$ test functions among a set of $K'>K$ number of test functions. That is to say, we can, instead of defining $K$ local hat functions and training on all of them, define $K'$ local hat functions, which discretized the domain better, but still keep training on only $K$ of them, randomly chosen on each epoch. Stochastically, the network will be optimized for all the test functions.

    \item Investigating the evolution of the solution throughout the training, we could see that with some configurations, the network manages to capture either the real or the imaginary part of the solution very well, but lacks accuracy in the other part of the solution. The evolution of the solution with random initialization and with least-squares initialization are available in \autoref{app:evolutions}. Regarding this observation, we suggest that training two networks for each part (imaginary, real) of the solution could be beneficial.

\end{itemize}


%
%

\chapter{State space generation algorithms} \label{SEC:SSGen}

\section{Finite state spaces} \label{SEC:finite-state-spaces}

A discrete-state model expressed in a high-level formalism implicitly specifies:
\begin{itemize}
\item $\widehat{\sset}$, the set of \emph{potential states} describing
the ``type'' of states.
\item $\initstate \in \widehat{\sset}$, the \emph{initial state}
(we use boldface lower-case roman characters to denote states).
\item $\nset : \widehat{\sset} \rightarrow 2^{\widehat{\sset}}$, the
\emph{next--state function}, describing which states can be reached from
a given state in a single step.
In the formalisms we consider, $\nset$ is really expressed as a union
$\nset = \bigcup_{e \in \eset} \nset_{e}$, where~$\eset$ is a finite set
of \emph{events} and~$\nset_{e}$ is the next-state function associated
with event~$e$.
Thus, $\nset_{e}(\vi)$ is the set of states the system can enter when event $e$
occurs, or \emph{fires}, in state $\vi$.
In particular, $e$ is \emph{disabled} in $\vi$ if $\nset_e(\vi) = \emptyset$.
In the case of a Petri net with exponential timing, the set $\nset_{e}(\vi)$
can only be either empty or a singleton, since the firing of
a given transition $e$ in a given marking $\vi$
deterministically leads to a unique marking $\vj$.
However, nondeterminism can be achieved, i.e., $\nset_{e}(\vi)$ can contain
multiple elements, by using immediate transitions in the model.
\end{itemize}

From such a high-level model, we can build its
\emph{reachable state space}, or \emph{reachability set},
$\sset \subseteq \widehat{\sset}$, as
the smallest set containing the initial state $\initstate$ and
closed with respect to~$\nset$:
$$
   \sset = \{\initstate\} \cup \nset(\initstate) \cup
   \nset(\nset(\initstate)) \cup \cdots = \nset^{\ast}(\initstate).
$$
The \emph{reachability graph} is instead the directed graph having $\sset$
as nodes, and an arc from $\vi$ to $\vj$ iff $\vj \in \nset(\vi)$.
If we are interested in the event identities, we label the arc
with the event(s) $e$ for which $\vj \in \nset(\vi)$.

When discussing discrete-state systems, it is natural to consider
a state as a collection of \emph{substates}, thus
the potential state space $\widehat{\sset}$ is the
cross-product of some fixed number $K$ of \emph{local state spaces},
$$
  \widehat{\sset} = \sset_K \times \sset_{K-1} \times \cdots \times \sset_1 .
$$

Unlike many other tools, {\smart} does not require the user to
\emph{explicitly} define the exact composition of these local state spaces.
For example, a marking $\vi$ of a Petri net is simply a vector of $K$
natural numbers, where $K$ is the number of places in the net,
but {\smart} simply needs to know this number $K$, i.e., it only needs to know
that $\vi \in \Naturals^K$, not what the possible number of tokens
in each place can be.

This is more user-friendly than requiring the
specification of redundant information (after all, the Petri net itself
implicitly defines the elements of the sets $\sset_k$ for $K \geq k \geq 1$),
but it also carries with it a responsibility:
it is up to the user to make sure that the state space is finite,
since $\widehat{\sset}$ is not anymore the cross-product of
$K$ finite (explicitly enumerated by the user) sets.
Of course, if desired, the user can still explicitly bound the local
state spaces, without having to list them.
For example, in a Petri net, this can be achieved by introducing inhibitor
arcs or complementary places that limit the number of tokens
in the places;
however, this is often not needed, as many interesting models
are naturally finite, as they are covered by invariants.

State space exploration algorithms are available in {\smart} as
functions that can be used to build measures.
We use the \Code{pn} formalism for our examples, but
the predefined functions we describe are applicable to any other
formalism in {\smart}.
The applicable algorithms are independent of the data structure used to
represent the reachability set $\sset$, although the choice for the
latter can greatly affect their memory and runtime requirements.
The available functions are:

\begin{itemize}

\item
\Code{bigint num\_states()}\\
Builds $\sset$ if necessary, and returns the size of $\sset$.
Whenever $\sset$ is built,
if \Code{procgen} is set for options \Code{Debug} and \Code{Report},
then debugging and performance information (respectively)
for the state space generation engine are displayed.

\item
\Code{void show\_states()}\\
Builds $\sset$ if necessary, and explicitly lists the elements of $\sset$.
Option \Code{MaxStateDisplay} specifies the maximum number of states
to be displayed,
and option \Code{StateDisplayOrder} dictates the order in which states
are displayed.

\item
\Code{bigint num\_arcs()}\\
Builds the reachability graph if necessary,
and returns the number of arcs in the graph.
{\smart} builds a state-to-state transition matrix to evaluate this function,
hence duplicate arcs that lead from one state to the same state
due to different events are not counted multiple times.
Whenever the reachability graph is built,
if \Code{procgen} is set for options \Code{Debug} and \Code{Report},
then debugging and performance information (respectively)
for the reachability graph generation engine are displayed.

\item
\Code{void show\_arcs()}\\
Builds the reachability graph if necessary,
and displays it.
Option \Code{MaxArcDisplay} specifies the maximum number of arcs
to be displayed,
while options \Code{DisplayGraphNodeNames}
and \Code{GraphDisplayStyle} control the style of the graph output.
\end{itemize}

The following is a {\smart} program that computes the number of reachable states
for the classical model of the dining philosophers, where the number \Code{N}
of philosophers is an input parameter:
%
\lstinputlisting[firstline=3]{examples/phils_states.sm}
%

\section{Algorithms and data structures}

When the generation of the state space is required,
{\smart} uses the value of the \Code{ProcessGeneration} option
to determine the family of algorithms to choose from:
\begin{itemize}
  \item \Code{EXPLICIT}:\\
  States are stored explicitly, using {\smart}'s specialized state storage
  schemes.
  Explicit algorithms perform a breadth-first search where
  each state is stored and explored individually: for each
  newly-discovered state, every event is tested to see whether it is enabled
  and, if so, it is fired to obtain new states
  \cite{1997Tools-Storage}.
  The option \Code{ExplicitStateStorage} specifies which dictionary
  data structure to use for determining if a discovered state
  is already in $\sset$ or not.

  \item \Code{MEDDLY}:\\
  States are stored symbolically, using Meddly to support
  the decision diagram representation.
  \begin{release}
  Currently,
  this requires the user to specify a \emph{partition} for the model,
  which dictates how state variables are collected and mapped
  to decision diagram variables.
  \end{release}
  The option \Code{MeddlyProcessGeneration} specifies
  the generation algorithm (including explicit)
  to use with Meddly.
  \begin{release}
  Some of these algorithms are not implemented yet,
  while others have prototype implementations.
  \end{release}
  Several other options exist to set parameters of Meddly.
\end{itemize}

\TBD{Expand this?}

\section{Defining a partition of the model} \label{SEC:partition}

\index{partition!defining|(}

A partition must be defined when the value of the \Code{ProcessGeneration} option
is set to \Code{MEDDLY}.
Using \Code{EXPLICIT} methods is still possible when a partition is
specified, since {\smart} simply ignores this information in this case.

To illustrate how to partition a model, consider the dining philosophers
example of the previous section.
We can add the statement
\begin{lstlisting}
  partition({Idle[i], WaitL[i], WaitR[i], HasL[i], HasR[i], Fork[i]} : i+1);
\end{lstlisting}
inside the first \Code{for} loop.
This assigns the places belonging to philosopher~$i$ and fork~$i$ to
partition number~$i+1$, that is, it assigns one philosopher per submodel.
When symbolic methods are used,
{\smart} will use the highest-numbered partition as the topmost,
i.e., the one corresponding to the root of the MDD,
while the lowest-numbered partition will be the bottom level of the MDD.

Note that, when directly assigning the places to a given class of the
partition using an integer, this can be done in any order:
the statements
\begin{lstlisting}
partition(p:4); partition(q:7); partition(r:4);
\end{lstlisting}
are legal, in that order; however, it is illegal to assign the same
place \Code{p} to multiple classes, as in
\begin{lstlisting}
partition(p:4); partition(p:7);
\end{lstlisting}

Instead of specifying the index of the class explicitly, we can let
{\smart} assign indices, thus we could have used the statement
\begin{lstlisting}
partition({Idle[i], WaitL[i], WaitR[i], HasL[i], HasR[i], Fork[i]});
\end{lstlisting}
to define the partition instead.
When this ``index-less'' mode is used, {\smart} puts the first
class defined for the partition in the topmost level, the second one in
the second topmost, and so on.
If both partition definitions are used in the same model,
e.g.,
\begin{lstlisting}
partition({p, q}); partition({r, t}:7);
\end{lstlisting}
then {\smart} will place all the indexed partitions above the index-less ones.

We conclude this section by reminding the user that,
as in any BDD-based technique, the variable ordering (and, in our case,
the partition choice as well)
is crucial to the efficiency of the state-space construction algorithm.
The specification of a ``good'' partitioning is often a challenging
task that is for the moment left to the user;
{\smart} can only check whether a partition is ``legal''.

\index{partition!defining|)}


\chapter{Conclusions} \label{Conclusion}
There are various algorithms for outlier detection in datasets. Two of them are Local Outlier Factor and One Class Support Vector Machine. Both algorithms have been used to detect outliers in a credit card fraud dataset. These algorithms require labeled normal data, which is essential in semi-supervised learning algorithms. The algorithms assigned a binary label to the observations, where 1 refers to normal observation and -1 refers to an outlier. Local Outlier Factor and One Class Support Vector Machine have different ways to identify outliers. They use \em{density} and \em{distance} of the observations respectively.

This research proposes a comparison of these two algorithms for the outlier detection process. Both algorithms reach different percentages of performance. But One Class Support Vector Machine performs better than Local Outlier Factor in the given dataset.

Comparison of these algorithms has been done by using the results of a confusion matrix, precisely, True Positive Rate, and False Positive Rate. The algorithm that has a higher True Positive Rate and a False Positive Rate is the better one.

The Local Outlier Factor True Positive Rate is from 80.4$\%$ to 70.1$\%$, while the False Positive Rate is from 90.1$\%$ to 90.2$\%$ and the Area Under the Curve is from 35$\%$ to 51$\%$. The One Class Support Vector Machine algorithm shows different behaviour, the True Positive Rate is from 98.1$\%$ to 97.8$\%$, False Positive Rate is from 21.6$\%$ to 14.6$\%$ and the Area Under the Curve is from 93$\%$ to 94$\%$. To conclude, the algorithms have shown different Positive Positive Rate, False Positive Rates and areas of Receiver Operator Characteristic. One Class Support Vector Machine displays a better performance on detecting outliers.

The current results are obtained by applying the Local Outlier Factor algorithm and One Class Support Vector Machine in some transactions that selected randomly from the dataset. Therefore the behaviour of the algorithms may change with different transactions. Further work could be done by applying the algorithms on the whole dataset which will give more precise results. Applying these algorithms in different datasets to see if the results convergence or divergence will give a better understanding of algorithms. Adding more machine learning outlier detection algorithms will help to know which one is better to apply in Big Data.


\documentclass[a4paper,12pt]{report}

\usepackage{amsmath,amsfonts,mathtools}
\usepackage{hyperref}

\begin{document}
\title{AER210 Abridged}
\author{Aman Bhargava}
\date{September 2019}
\maketitle

\tableofcontents

\pagebreak

\chapter{Review: Stuff to Have Memorized}
%TODO: Populate this with stuff from MAT195
\section{Trig Functions and Derivatives}
\def\arraystretch{2}%
\begin{tabular}{cc}
$ \frac{d}{dx}sin(x) = cos(x) $ & $ \frac{d}{dx}csc(x) = -csc(x)cot(x) $ \\
$ \frac{d}{dx}cos(x) = -sin(x) $ & $ \frac{d}{dx}sec(x) = sec(x)tan(x) $ \\
$ \frac{d}{dx}tan(x) = sec^2(x) $ & $ \frac{d}{dx}cot(x) = -csc^2(x) $ \\
\end{tabular}

\section{Inverse Trig Derivatives}
\def\arraystretch{2}%
\begin{tabular}{cc}
$ \frac{d}{dx}sin^{-1}(x) = \frac{1}{\sqrt(1-x^2)} $ \\
$ \frac{d}{dx}cos^{-1}(x) = \frac{-1}{\sqrt(1-x^2)} $ \\
$ \frac{d}{dx}tan^{-1}(x) = \frac{1}{1+x^2} $ \\
\end{tabular}

\section{How to complete the Square}
\begin{enumerate}
\item Put $ax^2 + bx$ in brackets and forcefully factor out the $a$
\item Add $ ( \frac{b}{2})^2$ to the inside of the brackets and subtract it from the outside (you got it)
\item Factor and be happy that you've completed the square;
\end{enumerate}

\section{Trig Angle Sums}
\begin{enumerate}
\item $sin(A+B) = sin(A)cos(B) + cos(A)sin(B)$
\item $cos(A+B) = cos(A)cos(B) - sin(A)sin(B)$
\item $sin(A-B) = sin(A)cos(B) - cos(A)sin(B)$
\item $cos(A-B) = cos(A)cos(B) + sin(A)sin(B)$
\end{enumerate}

\section{Identities Assorted}
\begin{eqnarray}
sin^2 (x) = 1/2 (1-cos(2x)) \\
cos^2 (x) = 1/2 (1+cos(2x)) \\
sinxcosx = 1/2 sin2x
\end{eqnarray}

\begin{eqnarray}
sin A cos B = 1/2 [sin(A-B) + sin(A+B)] \\
sin A sin B = 1/2 [cos(A-B) - cos(A+B)] \\
cos A cos B = 1/2 [cos(A-B) + cos(A+B)]
\end{eqnarray}

\begin{eqnarray}
\frac{d}{dx} csc(x) = -csc(x)cot(x) \\
cot^2(x) = csc^2(x) - 1 \\
\frac{d}{dx} cot(x) = -csc^2(x)
\end{eqnarray}

\chapter{In-Class Review}
\section{Vectors \& Vector Functions}

\begin{itemize}
\item Vector = magnitude + direction
\item If the origin of the vector is the origin of the coordinate system, it's a position vector.
\item Dot product:
$\vec{a} \cdot \vec{b} = \vec{a}_1 \cdot \vec{b}_1 + ... + \vec{a}_n \cdot \vec{b}_n$
\item Cross product: $\vec{a} \times \vec{b} = det(i, j, k; \vec{a}^{T}; \vec{b}^{T})$
\item Cross product is the area of the paralellogram traced out by the two vectors.
\item Scalar triple product: $\vec{a} \cdot (\vec{b} \times \vec{c})$, produces a scalar, represents the volume of the parallelepiped formed by the three vectors.
\item To get the derivative of a vector function, simply  take the derivative of each of the internal functions and package them into a new vector function.
\end{itemize}

\section{Arc Length}
\subsection{One-Variable Functions}
$$L = \int_a^b \sqrt{1+[f'(x)]^2} dx$$

\subsection{Parametric Functions}
Let $y(t)$ and $x(t)$ describe a parametric function in 2 dimensions. Then the arc length would be:
$$L = \int_a^b \sqrt{[x'(t)]^2 + [y'(t)]^2} dx$$

\subsection{Vector Funtions}
Let $\vec{r}(t)$ describe a vector function that converts a scalar $t$ into a vector. Then the arclength function would be:
$$L = \int_a^b |\vec{r}(t)| dt$$

\subsection{Reparamerizing with respect to Arc Length}
\paragraph{What is this? } Let there be a vector function $\vec{r}(t)$ and its correponding arc length function
$s(t)$. Since $s$ is strictly increasing, we can safely \textbf{reparameterize} $\vec{r}(t)$ to be
$\vec{r}(s(t)) \to \vec{r}(s)$.
\paragraph{Why would you want to do this? } This type of reparameterization is useful because now we do not have to
rely on any particular coordinate system.
\paragraph{Steps to Reparameterizing}
\begin{enumerate}
\item Find $s(t) = \int_a^b |\vec{r}(u)| du$.
\item Put $s$ in terms of $t$.
\item Substitute the expression found in part 2 in the original $\vec{r}(t)$.
\end{enumerate}

\section{Partial Derivatives}
\subsection{Functions of Several Variables}
\paragraph{A function of two variables } transforms each pair of Reals $(x, y)$ in a given set to a single real number. The given set is the domain, and the set of reals that the pair is transformed to is the range.

\paragraph{Level functions } are functions that have $f(x, y) = k$ for given ranges of $(x, y)$

\paragraph{Functions of 3 or more variables } are pretty easy to extrapolate from functions of two variables, tbh.

\subsection{Limits and Continuity with Functions of Several Variables}
\subsubsection{Limits}
\paragraph{Definition of limit } with many variables:
$$\lim_{(x, y) \to (a, b)} f(x, y) = L$$
if for every number $\epsilon > 0$ there is a corresponding number $\delta > 0$ s.t.
if $0 < \sqrt{(x-a)^2 + (y-b)^2} < \delta$ then $|f(x, y) - L | < \epsilon$

\paragraph{How to find: }
Regard the non-mentioned variable in the notation as a constant and differentiate with respect to the mentioned variable.

\subsection{Higher Partial Derivatives}
$$(f_x)_y = f_{xy} = f_{12} = \frac{\partial}{\partial y} (\frac{\partial f}{\partial x}) = \frac{\partial ^2 f}{\partial y \partial x} = \frac{\partial z}{\partial y \partial x}$$

\subsubsection{Clairaut's Theorem}
If $f_{xy}$ and $f_{yx}$ are both \textbf{defined} and \textbf{continuous} on disk $D$ then:
$$f_{xy}(a, b) = f_{yx}(a, b)$$

\section{Gradient}
Think of the gradient like an operator that applies to functions of many variables (functions of vectors). The $\nabla$
just calculates the partial derivatative of the function with respect to each of its input variables and puts it
into a vector.

$$\nabla f(x, y) = [\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}]$$

Or, more generally for a function $f(\vec{x})$, $$\nabla f(\vec{x}) = [\frac{\partial f}{\partial x_1}, ... ,
\frac{\partial f}{\partial x_n}]$$

\section{Chain Rule with Many Variables}
Let there be a function $f(\vec{x})$. Let $\vec{x}$ of length $n$ be a function of $\vec{t}$ of length $m$.
We take the partial derivative of $f$  with respect to $t_i$ by the following:

$$\frac{\partial f}{\partial t_i} = \frac{\partial f}{\partial x_1} \frac{\partial x_1}{\partial t_i} +
... + \frac{\partial f}{\partial x_n} \frac{\partial x_n}{\partial t_i} $$

\chapter{Multiple Integrals}
Pretty much the same as regular integrals, you just do two. You can apply them to volumes under surfaces.

\section{Basic Meaning and Solving}

\section{Leibniz Integral Rule (Differentiability of Integral with Respect to Parameter)}
The Leibniz integral rule simply lets you more easily take the derivative of the integral of a multivariable
function where the variable you are integrating with respect to is not the same as the variable you are taking
the derivative with respect to.
$$\frac{d}{dx} \int_a^b f(x, t) dt$$

\subsection{Constant Bounds of Integration}
When you are integrating from one constant to another $[a, b]$, the result is quite simple and elegant.
$$\frac{d}{dx} \int_a^b f(x, t) dt = \int_a^b \frac{\partial f}{\partial x} dt$$

\subsection{Derivation}
Let's write $\frac{d}{dx} \int_a^b f(x, t) dt$ in terms of the definition of the derivative:
\begin{align*}
& = \frac{\int_a^b f(x +\Delta x, t)dt - \int_a^b f(x+\Delta x, t)dt}{\Delta x} \\
& = \frac{\int_a^b f(x, t) dx + \int_a^b \frac{\partial f}{\partial x} dt \Delta t dx - \int_a^b f(x, t) dt}{\Delta x} \\
& = \int_a^b \frac{\partial f}{\partial t} dx \\
\label{Constant Bounds}
\end{align*}

\subsection{Variable Bounds of Integration}
Final result:
$$\frac{d}{dx} \int_{a(t)}^{b^t} f(x, t) dt = \int_{a(t)}^{b(t)} \frac{\partial f}{\partial x} dt - f(a, t)\frac{da}{dt} + f(b, t)\frac{db}{dt}$$

\section{Polar Coordinates with Multiple Integrals}
Recall $r^2 = x^2 + y^2$, $x = rcos(\theta)$, $y = rsin(\theta)$

\paragraph{A polar rectangle} is of the form $$R = {(r, \theta) | a \leq r \leq b, \alpha \leq \theta \leq \beta}$$

\paragraph{Basic form of double integral in polar coordinates: }
$$\iint_R g dA = \int_{\alpha}^{\beta} \int_a^b g(r, \theta)\,dr\,d\theta$$

\subsection{Change to Polar Coordinates in Double Integrals}
$$\iint_R f(x, y) dA = \int_{\alpha}^{\beta}\int_a^b f(rcos(\theta), rsin(\theta))*r \,dr\,d\theta$$
Make sure not to forget the $r$ in the integral!

\subsection{Variable Bounds of Integration for $r$}
$$D = {(r, \theta) | \alpha \leq \theta \leq \beta, h(\theta) \leq r \leq g(\theta)}$$
Then:
$$\iint_D f(r, \theta) = \int_{\alpha}^{\beta} \int_{h(\theta)}^{h(\theta)} f(r, \theta)*r \, dr \, d\theta$$

\section{Applications of Multiple Integrals}
The obvious application of multiple integrals is to compute volume. But, like with regular
integrals, there are a lot more ways you can apply them (think back to centre of mass,
arc length, etc).

\subsection{Density and Mass}
\paragraph{Lamina } is a plan of mass and surface density. With double integrals,
we can now calculate mass of a lamina with \textbf{variable} density!

$$\rho(x, y) = \lim \frac{\Delta m}{\Delta A}$$
$$\rho(x, y) = \frac{\partial m}{dx dy}$$

\paragraph{Calculating Mass: }
$$m = \iint_D \rho(x, y) dA$$

\subsection{Moments of Intertia}

\paragraph{Moment $M_x$} is the distance of the centre of mass from the $x$-axis times
the mass. Likewise with $y$.

\paragraph{Centre of Mass } is at $(\bar{x}, \bar{y})$ where $m\bar{x} = M_y$ and
$m\bar{y} = M_x$.

If you're confused as to why the $\bar{x}$ and $M_y$ are related, just remember that
the moment about the $y$-axis ($M_y$) is defined as the \textbf{distance from the
y-axis to the centre of mass} times the mass of the lamina. That distance is the $\bar{x}$
value!

\subsection{Moments of Intertia}

Recall that $I = mr^2$ for particles.

$$I_x = \iint_D y^2 \rho(x, y)dA$$
$$I_y = \iint_D x^2 \rho(x, y)dA$$

$$I_o = I_x + I_y = \iint_D (x^2 + y^2) \rho(x, y)dA$$
Makes sense because $x^2+y^2$ is the distance of the chunk from the
origin.

\subsubsection{Radius of Gyration}
\paragraph{About Arbitrary Axis: } $R$ is the radius of gyration if $mR^2 = I$. Basically, if mass were concentrated at $R$,
then it would have the same moment of inertia as the original lamina.

\paragraph{About the Origin: } $(\bar{\bar{x}}, \bar{\bar{y}})$ is the point of gyration about the origin.

\subsection{Probability}
The probability function of a random variable $x$ is:
$$f(x) > 0\,\forall \, x, \int_{-\infty}^{\infty} f(x) dx = 1$$

\paragraph{Probability of $a < x < b$: } $\int_a^b f(x) dx$

\paragraph{2-D Probability Function: } Chance of $(x, y) \in D$ is $\iint_D f(x, y) dA$.

Note that $\iint_{\mathbb{R}^2} f(x, y) dA = 1$

\subsubsection{Expected Values}
For single variable $x$ with probability density function $f$, the expected value is the
average value of $x$ across many trials.

$$\mu = \int_{-\infty}^{\infty} x f(x) dx$$

Makes sense because you're just summing the products of the chance of a particular $x$
and the value of that $x$ for all $x$.

\paragraph{Multi-Variable Functions: }
$$\mu_1 = X_{mean} = \iint_{\mathbb{R}^2} x f(x, y) dA$$
$$\mu_2 = Y_{mean} = \iint_{\mathbb{R}^2} y f(x, y) dA$$

\paragraph{Normal Distribution: } $$f(x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-(x-\mu )^2/(2\sigma^2)}$$

\section{Surface Area}

\paragraph{How it's derived: } If you divide the surface in to an $m$ by $n$ grid, the
surface area of each piece is the area of that parallelogram. That can be found by
the cross product of the two vectors that define two sides of the parallelogram. Those
vectors are just the width and length of the piece times the partial derivatives along
that particular axis.

$$A(S) = \iint_D \sqrt{(\frac{\partial f}{\partial x})^2
                     + (\frac{\partial f}{\partial y})^2 + 1} dA$$

$$A(S) = \iint_D \sqrt{(\nabla f(x, y))^2 + 1} dA$$

The $+ 1$ term is there because it's $\frac{\partial f}{\partial f}$.

\section{Triple Integrals}
\subsection{Type 1}
Lies between continuous functions of $f(x, y)$

$$E = {(x, y, z) | (x, y) \in D, u_1(x, y) \leq z \leq u_2(x, y) }$$

Then $$\iiint_E f(x, y, z) dV = \iint_D [\int_{u_1(x, y)}^{u_2(x, y)} f(x, y, z) dz] dA$$

\subsection{Type 2}
Lies between continuous functions of $f(y, z)$

$$E = {(x, y, z) | (y, z) \in D, u_1(y, z) \leq x \leq u_2(y, z) }$$

Then $$\iiint_E f(x, y, z) dV = \iint_D [\int_{u_1(y, z)}^{u_2(y, z)} f(x, y, z) dx] dA$$

\subsection{Type 3}
Lies between continuous functions of $f(x, z)$
$$E = {(x, y, z) | (x, z) \in D, u_1(x, z) \leq y \leq u_2(x, z) }$$

Then $$\iiint_E f(x, y, z) dV = \iint_D [\int_{u_1(x, z)}^{u_2(x, z)} f(x, y, z) dy] dA$$

\subsection{Applications of Triple Integrals}
\subsubsection{Obvious}
$$V = \iiint_E dV$$
$$m = \iiint_E \rho(x, y, z) dV$$

\subsubsection{Moments and CoM}
\paragraph{REMEMBER: } Moments are about a plane (e.g. $(x, y)$ plane) while moments of INERTIA are
about an axis (think about spinning vs. balancing - you balance along a plane, spin around an axis).

$$M_{yz} = \iiint_E x\rho(x, y, z) dV$$
etc.

$$I_x = \iiint_E (y^2 + z^2) \rho(x,y,z)dV$$

\section{Cylindrical Coordinates}
These work pretty much how you would think they'd work based on your knowledge of polar coordiantes in 2-D. Just stick a
conventional z-axis on the polar coordinate system and you have cylindrical coordinates.

$$x = r\cos\theta; \,\,\,\, y = r\sin\theta; \,\,\,\, z = z$$
$$\iiint_E f(x, y, z) dV = \iiint r f(r\cos(\theta), r\sin(\theta), z) dz\,dr\,d\theta$$

Don't forget that little $r$ coefficient in there!

\section{Spherical Coordinates}
Now we define points in 3-D space by the the following values:
\begin{enumerate}
\item $p$: the distance away from the origin (scalar)
\item $\theta$: angle from 0 of the point when it's projected onto the x-y plane.
\item $\psi$: The angle from the z-axis.
\end{enumerate}
So yeah, a bit of a pain in the ass to visualize and draw. At least it's useful when they give you literal spheres...

Anyway, here are the relationships between those newfangled varibles and our trusty old $x, y, z$ system:
\begin{enumerate}
\item $x = p\sin\phi\cos\theta$
\item $y = p\sin\phi\sin\theta$
\item $z = p\cos\phi$
\item $p = \sqrt{x^2+y^2+z^2}$
\end{enumerate}

Finally, here's the conversion formula for triple integrals:
$$\iiint_E f(x, y, z) dV = \iiint f(p\sin(\phi)\cos(\theta), p\sin(\phi)\sin(\theta), p\cos(\theta)) p^2\sin(\phi) dp\,d\theta\,d\phi $$

\section{Taylor Series with Two Variables}
...

\section{Jacobians}
Remember that time when $\int_a^b f(x) dx = \int_c^d f(g(u))\frac{dx}{du} du$? Well imagine if we could do that with
multiple integrals. How cool would that be?

Well guess what, we've been doing that for a while now! Whenever we transfer to cylindrical and spherical coordinates we're
basically just applying that same rule. The only new thing we have to learn is the equivalent of that $\frac{dx}{du}$ bit when
you have multiple independent variables.

\subsection{How it Works}
Let $(u, v) = (G(x, y), H(x, y))$ be our transformation we want to use to make the integral easier. And, let's say we
want to change the followinig:

$$\iint_V f(x, y) dx\,dy \to \iint_{V_2} f(u, v) du\,dv$$

All we gotta do is apply the same rule that we're used to using for u-substitutions, but the $\frac{dx}{du}$ term becomes:
$$
\frac{\partial (x, y)}{\partial (u, v)} = \det
\begin{vmatrix}
\frac{\partial x}{\partial u} & \frac{\partial x}{\partial v} \\
\frac{\partial y}{\partial u} & \frac{\partial y}{\partial v}  \notag
\end{vmatrix}
$$

You can kind of see where that Jacob guy was coming from. It sort of makes sense that
$\Delta A \approx |\frac{\partial (x, y)}{\partial (u, v)}| du\, dv$.

\subsection{How to Change your Variable with Jacobians}
$$\iint_R f(x, y) dA = \iint_S f(x(u, v), y(u, v))|\frac{\partial (x, y)}{\partial (u, v)}|du\,dv$$

This applies to n-dimensional integrals. You can double check that the transformations to cylindrical and spherical coordinates
work out if you want!

\chapter{Vector Calculus}
\section{Review: Vector Fields, Gradient Fields}
\subsection{Vector Fields}
Let $E$ be a subset of $\mathbb{R}^3$. A vector field on $\mathbb{R}^3$ is $F$ that assigns each
$(x, y, z) \in \mathbb{R}^3$ to a 3D vector $F(x, y, z)$.

\subsection{Gradient Fields}
The gradient operator is represented by $\nabla$ and can be viewed as the operator
$$
\begin{bmatrix}
\frac{\partial }{\partial x} \\
\frac{\partial }{\partial y} \\
\frac{\partial }{\partial z} \\
... \notag
\end{bmatrix}
$$

A scalar field transforms each point $(x, y, ...) \in \mathbb{R}^n$ into a real number in $\mathbb{R}$ (e.g. $f(x, y, ...) \to \mathbb{R}$).
A gradient field is a field of $n$-dimensional vectors that is made by taking the gradient of $f$ at each point $(x, y, z, ...)$ in
a given region $\nabla f(x, y, z, ...)$.

These differ from regular vector fields because \textbf{not every vector field can be made from the gradient of a scalar field}.
Think about that, it's really cool!

\section{Line Integrals}
From Stewart 16.2: It's like a regular integral, but we're integrating over a curve, not a
range of values.
\paragraph{Definition: } Let $f$ be defined along smooth curve $C$. Then the line integral of $f$
along $C$ is $$\int_C f(x, y) ds = \int_a^b f(x(t), y(t)) \sqrt{(\frac{dx}{dt})^2 + (\frac{dy}{dt})^2} dt$$

Visually it's just like taking the single integral along the path $<x(t), y(t)>$ as opposed to a straight line
along the $x$ or $y$ axis. More generally:

$$\int_C f(\vec{r}(t)) ds = \int_C f(\vec{r}(t))|r'(t)|dt$$

\subsection{Line Integrals with respect to $x$ and $y$}
You can also take the integral with respect to differences in $x$ and $y$ instead of with respect to
$s$ (the arc length).

$$\int_C f(x, y) dx = \int_a^b f(x(t), y(t)) x'(t) dt$$
$$\int_C f(x, y) dy = \int_a^b f(x(t), y(t)) y'(t) dt$$

Physically, that means that the `significance' of an infinitessimal part of the line is decided by the $x$ or $y$-component
of the piece of the line.

A shorthand for adding these together is as follows:
$$\int_C P(x, y) dx + \int_C Q(x, y) dy = \int_C P(x, y) dx Q(x, y) dy$$

\subsection{Path Independence}
It is not necessarily the case that $$\int_C f(x, y) dx = -\int_{-C}f(x, y) dx$$
However, when we take the line integral with respect to $s$ (path length), equality does hold!
$$\int_C f(x, y) ds = -\int_{-C}f(x, y)ds$$

\subsection{Line Integrals in Space}
This section pertains to 3D space. Let $C$ be a smooth space curve given by $<x(t), y(t), z(t)>$ for $a \leq t \leq b$

Now let $f$ be a function of three variables. The line integral of $f$ along $C$ is then
$$\int_C f(x, y, z) ds = \int_a^b f(x(t), y(t), z(t))\sqrt{(\frac{dx}{dt})^2+(\frac{dy}{dt})^2+(\frac{dz}{dt})^2} dt$$

Which is equivalent to:
$$\int_a^b f(r(t)) |r'(t)| dt$$ where $r(t)$ is the vector function $r(t)=<x(t), y(t), z(t)>$

\subsection{Line Integrals of Vector Fields}
Recall that the work done by force $f(x)$ on a particle from $a \to b$ along the $x$-axis is:
$$W = \int_a^b f(x) dx$$

Also recall that $W = F \cdot D$

Now we think of $F(x, y, z) = <p(x), q(y), r(z)>$ as a force field in 3-space. How do we compute the work done by
the force field on a moving particle?

$$W = \int_a^b F(r(t)) \cdot r'(t) dt = \int_C F \cdot T ds$$
Remembering that $F(r(t)) = F(x(t), y(t), z(t)) = [P, Q, R]$, so
$$\int_C F \cdot dr = \int_C P\,dx + Q\,dy + R\,dz$$

\subsection{Useful Tips}
The vector function that connects points $r_0, r_1$ is:
$$r(t) = (1-t)r_0 + t r_1 \,\,\, 0 \leq t \leq 1$$

\section{Fundamental Theorem of Line Integrals}
From 16.3 of Stewart.

Recall: the fundamental theorem of calculus states $\int_a^b F'(x) dx = F(b) - F(a)$ (a.k.a. the net change theorem).

Now let us generalize this to line integrals!

\paragraph{Fundamental Theorem of Line Integrals} Let $C$ be a smooth curve given by $r(t)$ and $f$
is a differentiable function of 2-3 variables with a continuous gradient on C.

$$\int_C \nabla f \cdot dr = f(r(b)) - f(r(a))$$

\subsection{Path Independence}
Here are some fun theorems to understand and have in the back of your mind:
\begin{enumerate}
\item \textbf{THM: } $\int_C \vec{F}\cdot\,d\vec{r}$ is path independent iff
$\int_C \vec{F}\cdot\,d\vec{r} = 0 \, \forall$ closed paths
\item \textbf{THM: } A vector field is conservative iff it is path independent.
$\exists$ a function $f$ such that $\nabla f = F$ iff conservative/path independent.
\item \textbf{THM: } if $\vec{F}(x, y) = [P(x, y)\hat{i}, Q(x, y)\hat{j}]$ is conservative,
then $$\frac{\partial P}{\partial y} = \frac{\partial Q}{\partial x}$$ assuming $P, Q$ have cts first derivatives and
the region is \textit{simply connected}.
\end{enumerate}

Some useful definitions:
\begin{enumerate}
\item A \textbf{simple curve} doesn't intersect itself between the ends.
\item A \textbf{simply connected region} has every simple, closed curve enclose only points in that region $D$.
\end{enumerate}

\section{Parametric Surfaces}
You can describe a 3D surface in terms of a 2D folded up surface:
$$\vec{r}(u, v) =
\begin{bmatrix}
x(u, v) \\
y(u, v) \\
z(u, v) \notag
\end{bmatrix}
$$

\subsection{How to Visualize and Plot}
Try to find relationships between the $x$, $y$, and $z$ funtions. Does $x^2 + z^2 = c$? If the function is $z = f(x, y)$ then the
parametric representation is:

$$\vec{r}(x, y) =
\begin{bmatrix}
x \\
y \\
f(x, y) \notag
\end{bmatrix}
$$

\subsection{Surface of Revolution}
Let's say you're given the function $f(x)$ and you want the surface of revolution in 3D.
$$
\vec{r}(x, \theta) =
\begin{bmatrix}
x \\
f(x)\cos\theta \\
f(x)\sin\theta \notag
\end{bmatrix}
$$

\subsection{Tangent Planes}
$$\vec{r}_v(u_0, v_0) =
\begin{bmatrix}
\frac{\partial x}{\partial v}(u_0, v_0) \\
\frac{\partial y}{\partial v}(u_0, v_0) \\
\frac{\partial z}{\partial v}(u_0, v_0) \notag
\end{bmatrix}
$$
$r_u$ is defined in a similar way. The normal vector to the surface is $r_u \times r_v$. From this you can determine the tangent plane at a point.

\subsection{Areas of Parametric Surfaces}
$$A(S) = \iint_D |\vec{r}_u \times \vec{r}_v| dA$$

\section{Surface Integrals}
These are the 2D equivalent of line integrals. For a parametric surface $\vec{r}(u, v) \to \mathbb{R}^3$ and a function $f(x, y, z) \to \mathbb{R}$:

$$\iint_S f(x, y, z) dS = \iint_D f(\vec{r}(u, v))|\vec{r}_u \times \vec{r}_v| dA$$

If you don't have a parametric version of your surface (you're stuck with $z = g(x, y)$), then just turn that into a parametric surface like we had before.

\subsection{Flux}
$$\iint_S \vec{F} \cdot d\vec{S} = \iint_S \vec{F} \cdot \hat{n} dS$$
where $\hat{n} = \frac{\vec{r}_u \times \vec{r}_v}{|\vec{r}_u \times \vec{r}_v|}$ is the normal vector to the surface at the point.

\paragraph{In $(u, v)$ space: } Flux is $$\iint_D \vec{F} \cdot (\vec{r}_u \times \vec{r}_v) dA$$

Basically, if you're still in normal 3D space for calculating flux, you need to multiply by the \textbf{unit normal vector} when you do your
dot product. If you're in $(u, v)$ space, then you can just multiply by the non-unit normal vector $(\vec{r}_u \times \vec{r}_v)$.

\subsection{Mass Flux}
For mass flux, then the vector field you integrate over is $\vec{v}(x, y, z)\rho(x, y, z)$

\section{Green's Theorem}
Provides a relationship between the line integral of a \textbf{simple closed curve $C$} and the \textbf{double integral of
region $D$ that is bounded by $C$}.

\paragraph{Note on orientation: } We assume a \textbf{SINGLE, COUNTERCLOCKWISE} traversal of $C$. The region is `to the left'
of $r(t)$ as it traverses $C$.

$$\int_C P\,dx+Q\,dy = \iint_D (\frac{\partial Q}{\partial x} - \frac{\partial P}{\partial y})dA$$

It's kind of the counterpart to $\int_a^b F'(x) dx = F(b)-F(a)$.

\section{Curl and Divergence}
\paragraph{Curl } is a measure of the rotation about a point in a vector field.
$$curl F = \nabla \times F =
\begin{vmatrix}
\hat{i} & \hat{j} & \hat{k} \\
\frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial }{\partial z} \\
P & Q & R \notag
\end{vmatrix}
$$

\paragraph{Divergence } is a measure of the amount of \textit{stuff} going outward from a point. Negative divergence means you have more stuff
going into the point (by measure of the vector field).
$$div F = \frac{\partial P}{\partial x} + \frac{\partial Q}{\partial y} + \frac{\partial R}{\partial z}$$

\subsection{Properties of Divergence and Curl}
\begin{enumerate}
\item If $F$ has continuous second order derivatives, then $div curl F = 0$.
\end{enumerate}

\subsection{Vector form of Green's Theorem}
$$\int_C F \cdot dr = \iint_D (curl\,F)\cdot k dA$$

In other words, you can infer the \textit{curliness} of a surface in a vector field based on a line interal of the edge.

\section{Divergence Theorem}
$E$ is simple solid region with boundary surface $S$ with outward orientation. $F$ is a vector field in that area
with continuous partial derivatives. Then
$$\iint_S \vec{F}\cdot d\vec{S} = \iiint_E div\,\vec{F} dV$$

\section{Stoke's Theorem}
If $S$ is an oriented piecewise-smooth surface with piecewise smooth closed boundary curve $C$ inside of
vector field $F$, then
$$\int_C \vec{F} \cdot d\vec{r} = \iint_S curl\,F\cdot d\vec{S}$$

\end{document}


% LyX 2.2.4 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english]{article}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=2.5cm,bmargin=2.5cm,lmargin=2.5cm,rmargin=2.5cm}
\setlength{\parskip}{\medskipamount}
\setlength{\parindent}{0pt}
\usepackage{url}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{babel}
\begin{document}

\title{UK Age and Space structured Covid-19 model}

\author{Chris Jewell, Barry Rowlingson, Jon Read}
\maketitle

\section{Concept}

We wish to develop a model that will enable us to assess spatial spread
of Covid-19 across the UK, respecting the availability of human mobility
data as well as known contact behaviour between individuals of different
ages.

A deterministic SEIR state transition model is posited in which individuals
transition from Susceptible to Exposed (i.e. infected but not yet
infectious) to Infectious to Removed (i.e. quarantined, got better,
or died).

We model the infection rate (rate of S$\rightarrow$E transition)
as a function of known age-structured contact from Polymod, known
human mobility between MSOAs (Middle Super Output Area), and Census-derived
age structured population density in regions across the UK.

Currently, this model is populated with data for England only, though
we are in the process of extending this to Scotland, Wales, and Northern
Ireland.

\section{Data}

\subsection{Age-mixing}

Standard Polymod social mixing data for the UK are used, with 17 5-year
age groups $[0-5),[5-10),\dots,[75-80),[80-\infty)$. Estimated contact
matrices for term-time $M_{tt}$ and school-holidays $M_{hh}$ were
extracted of dimension $n_{m}\times n_{m}$ where $n_{m}=17$.

\subsection{Human mobility}

2011 Census data from ONS on daily mean numbers of commuters moving
from each Residential MSOA to Workplace MSOA. MSOAs are aggregated
to Local Authority Districts (LADs) for which we have age-structured
population density. The resulting matrix $C$ is of dimension $n_{c}\times n_{c}$
where $n_{c}=152$. Since this matrix is for Residence to Workplace
movement only, we assume that the mean number of journeys between
each LAD is given by
\[
T=C+C^{T}
\]
with 0 diagonal.

\subsection{Population size}

Age-structured population size within each LAD is taken from publicly
available 2019 Local Authority data giving a vector $N$ of length
$n_{m}n_{c}=2584$, i.e. population for each of $n_{m}$ age groups
and $n_{c}$ LADs.

\section{Model}

\subsection{Connectivity matrix}

We assemble a country-wide connectivity matrices as Kronecker products,
such that
\[
M^{\star}=I_{n_{c}}\bigotimes M
\]
 and
\[
C^{\star}=C\bigotimes\bm{1}_{n_{m}\times n_{c}}
\]
giving two matrices of dimension $n_{m}n_{c}\times n_{m}n_{c}$. $M^{\star}$
is block diagonal with Polymod mixing matrices. $C^{\star}$ expands
the mobility matrix $C$ such that a block structure of connectivity
between LADs results.

\subsection{Disease progression model}

We assume an SEIR model described as a system of ODEs. We denote the
number of individual in each age-group-LAD combination at time $t$
by the vectors $\vec{S}(t),\vec{E}(t),\vec{I}(t),\vec{R}(t)$. We
therefore have
\begin{align*}
\frac{\mathrm{d\vec{S}(t)}}{dt} & =-\epsilon\frac{\vec{S}(t)}{N}-\beta_{t}\left[M^{\star}\vec{I}(t)+\beta_{2}w_{t}\bar{M}C^{\star}\frac{{\vec{I}(t)}}{N}\right]\frac{\vec{S}(t)}{N}\\
\frac{\mathrm{d}\vec{E}(t)}{dt} & =\epsilon\frac{\vec{S}(t)}{N}+\beta_{t}\left[M^{\star}\vec{I}(t)+\beta_{2}w_{t}\bar{M}C^{\star}\frac{{\vec{I}(t)}}{N}\right]\frac{\vec{S}(t)}{N}-\nu\vec{E}(t)\\
\frac{\mathrm{d}\vec{I}(t)}{dt} & =\nu\vec{E}(t)-\gamma\vec{I}(t)\\
\frac{\mathrm{d}\vec{R}(t)}{dt} & =\gamma\vec{I}(t)
\end{align*}
where $\bar{M}$ is the global mean person-person contact rate, and
$w_{t}$ is the total rail ticket sales in the UK expressed as a fraction
of the 2019 mean (a proxy for reduction in travel). Parameters are:

\[
\epsilon_{t}=\begin{cases}
\epsilon_{0} & \mbox{if}t<T_{b}\\
0 & \mbox{otherwise}
\end{cases}
\]

with $T_{b}$ a time beyond which imports to the UK are considered
to cease;
\begin{align*}
\beta_{t} & =\begin{cases}
\beta_{1} & \mbox{if }t<T_{L}\\
\beta_{1}\beta_{3} & \mbox{otherwise }
\end{cases}\\
\end{align*}
with $T_{L}$ the date of lock-down restrictions, $\beta_{1}$ a baseline
transmisison rate, and $\beta_{3}$ giving the ratio of post-lockdown
to pre-lockdown transmission; commuting infection ratio $\beta_{2}$;
latent period $\frac{1}{\nu}=4$days; and infectious period $\frac{1}{\gamma}$.
Typically, we assume that contact with commuters is $\beta_{2}=\frac{1}{3}$
of that between members of the same age-LAD combination assuming an
8 hour working day.

\subsection{Noise model}

Currently, and subject to discussion, we assume that all detected
cases are synonymous with individuals transitioning $I\rightarrow R$.
We assume the number of new cases in each age-LAD combination are
given by
\[
y_{ik}(t)\sim\mbox{Negative Binomial}\left(r,\phi(R_{ik}(t)-R_{ik}(t-1))\right)
\]

where $\phi$ is the case reporting fraction (i.e. proportion of infections
that are eventually detected) and $r$ is an overdispersion parameter.

\subsection{Inference}

We are interested in making inference on $\epsilon_{0},\beta_{1}$,
and $\gamma$. Priors are currently:
\begin{align*}
\epsilon_{0} & \sim\mbox{Gamma}(1,1)\\
\beta_{1} & \sim\mbox{Gamma}(1,1)\\
\beta_{3} & \sim\mbox{Gamma}(20,20)\\
\gamma & \sim\mbox{Gamma}(10,40)\\
r & \sim\mbox{\mbox{Gamma}(}0.1,0.1)
\end{align*}
specified to express relative ignorance about $\epsilon_{0}$ , $\beta_{1}$,
and $r$, strong information on $\beta_{3}$ and $r$, and a belief
that the infectious period should be approximately 4 days.

\subsection{Implementation}

The model is currently implemented in Python3, using Tensorflow 2.1
with the RK5 differential equation solver implemented in the \texttt{DormandPrince}
class provided by Tensorflow Probability 0.9. Code may be found at
\url{http://fhm-chicas-code.lancs.ac.uk/jewell/covid19uk}.

The code is a work in progress \textendash{} see README.

\section{Results}
\end{document}