\section{Introduction}
\label{introduction}

In scale-free networks (SF) the probability of a node being connected to $k$ others exhibits a power-law distribution $P(k) \propto k^{-\alpha}$, which is a topological property affecting and controlling their resilience or the measure of their functionality subject to disruptions \cite{Newman:2003da,Gao:2015fg,Bashan:2013cja,2000Natur.406..378A}. Examples of this class of SF are the Internet, power systems, and transportations networks, which are real-world networks shown to be robust when subject to random failures, yet, displaying a high vulnerability when subjected to cascading ones \cite{Bompard:2011cd,DuenasOsorio:2009ff,2016arXiv160904310M,Cohen:2001hf}.

In power systems, and unlike random failures which emerge locally, blackouts are severe events  associated with cascading behavior leading to global network collapse \cite{RosasCasals:2007td,Bompard:2009ga,Brummitt:2013jj,Daqing:2014bp,Albert:2004bw,Wang:2011js,Sole:2008cv}. Examples such as the one affecting the north-east in the US and eastern Canada in 2003 burdened the respective economies with ten billion dollars of direct costs \cite{Daqing:2014bp}. Such failures can be linked to either structural dependencies, where the damage spreads via structurally dependent connections, or functional overloads, where the flow goes through alternative paths leading to overloaded nodes. Thus, understanding the propagation of these failures becomes pivotal in developing and deploying protective and mitigating strategies.

Large power systems and real-world networks in general exhibit an exponentially increasing combinatorial number of failure nodes. Older works tackling contingency analysis relied on approximate power flow solutions, exhibiting only a simplified analysis of all combinatorial contingencies \cite{EjebeAl79,Ekwue91}. These approaches also levy an overwhelming computational burden that cannot be accomplished in real-time. Also, traditional approaches that employ the $N$-1 criterion are only able to investigate the ability of the transmission system to lose a power line or a power generator without causing an overload failure elsewhere. Examples of such approaches are employed by the North American Electric Reliability Corporation (NERC) \cite{JinAl10}, and obviously do not capture the overload failures that propagate through interactions among a system's physical components. It thus becomes necessary to be able to perform $N$-$x$ ($x \geq2$) contingency analysis to assess whether a system can withstand the failure of any two or more components. In several leading works such as \cite{2000Natur.406..378A,JinAl10, DaqingAl14}, the power grid, particularly the network of its transmission lines, is analyzed using graph algorithms such as betweenness centrality and shortest paths. In this paper, we build on the approach in  \cite{2000Natur.406..378A} using an Apache Spark implementation of topological vulnerability analysis of the Lebanese power grid subject to random  and cascading failures. Beyond the scholarly aspects of our proposed work, our analysis contributes towards precision-based policy making and disaster response in a region marred by wars and under-development.

\secondedited{
Using elements from distributed algorithm design as well as spatial analysis, our work extends the approach of \cite{2000Natur.406..378A} in analysing the vulnerability of transmission lines to the assessment of that of the whole Lebanese power grid, including its generation, distribution, and transmission lines. However, in contrast to this cited body of work, our approach yields a system that is fault-tolerant to hardware failures and ensures locality of reference to avoid data movement (e.g., network communications in case of distributed setting, and garbage collector and memory allocation in case of local setting). Despite that our input graph is not dramatically a ``big graph'', our use of Spark is meant to provide for the following: (1) scalability, represented by a solution that is shown to remain promising and effective when the input graph scales to such daunting sizes, when tackling either networks of networks or power plants in larger countries; (2) fault tolerance: if and when the input graph size increases, failures -- aka faults -- resulting from out-of-memory accesses and computations, message loss, network error, to name a few causes, would begin to appear very frequently as opposed to rarely, which, in the absence of a fault-tolerant platform, would require one to re-perform the whole computations. Spark provides for fault-tolerance using lineage techniques as well as efficient distributed in-memory computation. The choice of Spark also renders our system to be suitable for clusters of commodity computers with relatively slow and cheap interconnects, and susceptible to many machine failures.}
%
Our implementation is based on the BSP~\cite{bsp} model and employs serial graph algorithms on distributed data in SPMD mode (single program, multiple data). Our data parallel algorithm maintains parallel optimal-work, linear communication time, and a constant number of synchronisation barriers.”
%
%Our implementation is based on the bulk-synchronous parallel (BSP)~\cite{bsp} model and maintains optimal work, linear communication time, and a constant number of synchronisation barriers.
%By targeted attacks, we do not merely allude to acts of aggression on power generating centers, but also to increasing consumption associated with a diminished national carrying capacity following the refugees influx.

Our manuscript is organized as follows. In Sec. \ref{background}, we present an overview of related work that tackles power grid resilience analysis as well as implementations of it that runs on distributed systems, and we describe some intrinsics related to Apache spark for big data distributed processing. In Sec. \ref{methods}, we present our distributed graph algorithm that builds on the loosely centralized structure of the Lebanese power grid using Breadth First Search and Vertex Betweenness Centrality, orchestrated using four scenarios known in the literature \cite{2000Natur.406..378A}. Each of these scenarios simulates a unique temporal mode of removal of vertices.
%
\edited{Despite that our approach does not generalise to arbitrary networks, we believe there is merit in tackling the specifics related to the Lebanese power grid, especially after Lebanon has withstood decades of wars and a lack of infrastructure rebuilding. Still, our approach can generalize to networks of low connectivity such as that found in the Lebanese grid.}
%
In Sec. \ref{results}, we perform run-time analysis of our algorithm and obtain excellent scalability for some scenarios as the number of processing cores increases up to 16, a result which we attribute to the large number of connected components found in the Lebanese power grid and the fact that majority of connected components have a relatively small cardinality. In fact, scalability towards $32$ nodes improves when experimenting with replicas of the power grid graph that are double and quadruple the original size. We demonstrate the loss in connectivity in the power grid associated with each scenario, and obtain that the Lebanese power grid exhibits a relatively strong resilience thanks to its decentralised structure. We also perform a spatial correlation analysis of failures using the geocoding of vertices that lead to $90\%$ loss in connectivity. Our results reveal that failures in the power grid are spatially long-range correlated, and that correlations decay with distance. In Sec. \ref{conclusion}, we conclude with remarks around the impact of our work.%, particularly, as it relates to directing policy making. should current trends of refugee influx continue. It is our hope that our work can help answer questions such as: what factors will affect future observations at different locations on Lebanese soil? and what can be done to better manage our dwindling vital systems?


\subsubsection{Duration Adjusted CMS Leg Data}
\label{ss:duration_adjusted_cmslegdata}

Listing \ref{lst:duration_adjusted_cmslegdata} shows an example for a leg of type DurationAdjustedCMS.

\begin{listing}[H]
%\hrule\medskip
\begin{minted}[fontsize=\footnotesize]{xml}
      <LegData>
        <LegType>DurationAdjustedCMS</LegType>
        <Payer>false</Payer>
        <Currency>EUR</Currency>
        <Notionals>
          <Notional>21000000</Notional>
        </Notionals>
        <DayCounter>Simple</DayCounter>
        <PaymentConvention>Following</PaymentConvention>
        <PaymentLag>0</PaymentLag>
        <PaymentCalendar>TARGET</PaymentCalendar>
        <DurationAdjustedCMSLegData>
          <Index>EUR-CMS-20Y</Index>
          <Duration>10</Duration>
          <Spreads>
            <Spread>0</Spread>
          </Spreads>
          <Gearings>
            <Gearing>1</Gearing>
          </Gearings>
          <IsInArrears>false</IsInArrears>
          <FixingDays>2</FixingDays>
          <Caps>
            <Cap>0.015</Cap>
          </Caps>
          <NakedOption>true</NakedOption>
        </DurationAdjustedCMSLegData>
        <ScheduleData>
          <Rules>
            <StartDate>2019-12-31</StartDate>
            <EndDate>2023-12-31</EndDate>
            <Tenor>3M</Tenor>
            <Calendar>EUR</Calendar>
            <Convention>ModifiedFollowing</Convention>
            <TermConvention>ModifiedFollowing</TermConvention>
            <Rule>Backward</Rule>
          </Rules>
        </ScheduleData>
      </LegData>
\end{minted}
\caption{Duration Adjusted CMS leg data}
\label{lst:duration_adjusted_cmslegdata}
\end{listing}

The DurationAdjustedCMSLegData is identical to the CMSLegData block (see \ref{ss:cmslegdata}). In addition to this it
contains a field defining the duration adjustment:

\begin{itemize}
\item Duration [Optional]: A non-negative whole number $n$ that defines the duration adjustment for the coupons. If $\gamma$ is
  the underlying CMS index fixing for a coupon the duration adjustment $\delta$ is defined as
  $$\delta = \sum_{i=1}^n \frac{1}{(1+\gamma)^i}$$
  If $n$ is zero or the duration is not given, $\delta$ is defined as $1$, i.e. no adjustment is applied in this case and the
  coupon will be an ordinary CMS coupon. The coupon amount $A$ for a duration adjusted coupon with a spread $s$, a gearing
  $g$, a cap $c$, a floor $f$ and with nominal $N$ and accrual fraction $\tau$ is given by:
  $$A = \delta \cdot N \cdot \tau \cdot \max( \min( g \cdot \gamma + s, c ), f ) $$

  Allowable values:  A non-negative whole number.

\end{itemize}


\documentclass[12pt, a4paper]{article}
\usepackage[margin=1.25in]{geometry}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{float}
\usepackage{listings}
\usepackage{caption}
\usepackage[shortlabels]{enumitem}

\setlength\parindent{0pt}
\newcommand{\code}{\lstinline[basicstyle=\small]}
\lstset{
    language=Python,
    basicstyle=\scriptsize
}

\title{EE2703: Applied Programming Lab \\ \Large Assignment 5: Laplace Equation}
\author{Soham Roy \\ \normalsize EE20B130}
\date{\today}
\begin{document}

\maketitle % Insert the title, author and date

\bigskip
\section{Introduction}
The currents in a resistor will be evaluated. The currents depend on the shape of the resistor.
The part of the resistor likely to get hottest is also visualised. \\
\medskip

A wire is soldered to the middle of a copper plate and its voltage is held at 1 Volt. One side of the plate is
grounded, while the remaining are floating. The plate is 1 cm by 1 cm in size.

By the conductivity:
\begin{equation*}
    \vec{j} = \sigma \vec{E}
\end{equation*}

As the Electric field is the gradient of the potential:
\begin{equation*}
    \vec{E} = - \nabla \phi
\end{equation*}

Continuity of charge yields:
\begin{equation*}
    \nabla \cdot \vec{j} = - \frac{\partial \rho}{\partial t}
\end{equation*}

Combining these equations:
\begin{equation*}
    \nabla \cdot (- \sigma \nabla \phi) = - \frac{\partial \rho}{\partial t}
\end{equation*}

Assuming that the resistor consists of a material of constant conductivity:
\begin{equation*}
    \nabla^2 \phi = \frac{1}{\sigma} \frac{\partial \rho}{\partial t}
\end{equation*}

For DC currents the RHS is zero, thus:
\begin{equation*}
    \nabla^2 \phi = 0
\end{equation*}

\section{Subquestions}
\subsection{Import the Libraries}
The following libraries have been imported:
\begin{lstlisting}
    import sys
    import argparse
    import numpy as np
    import matplotlib.pyplot as plt

    from scipy.linalg import lstsq
\end{lstlisting}
\code{sys} and \code{argparse} are used to parse the command line arguments. \\
\code{numpy} and \code{matplotlib} are used for numerical computation and plotting. \\
\code{scipy.linalg.lstsq} is used to extract best fits for the linear system.

\subsection{Define the Parameters}
The following parameters have been defined:
\begin{lstlisting}
    Nx, Ny, radius, Niter = parse_cmdline_args()  # define the parameters
\end{lstlisting}
Here, \code{parse_cmdline_args()} is a function that parses the command line arguments and returns
the parameters as either user inputs or their default values. It employs the following code:
\begin{lstlisting}
    try:  # try converting directly to values
        return [
            int(sys.argv[1]) if len(sys.argv) >= 2 else 25,
            int(sys.argv[2]) if len(sys.argv) >= 3 else 25,
            int(sys.argv[3]) if len(sys.argv) >= 4 else 8,
            int(sys.argv[4]) if len(sys.argv) >= 5 else 1500,
        ]
    except ValueError:  # otherwise, parse keyword arguments
        parser = argparse.ArgumentParser(description="Resistor simulation")

        parser.add_argument(
            "-x", "--Nx", type=int, default=25, help="size along x axis"
        )
        parser.add_argument(
            "-y", "--Ny", type=int, default=25, help="size along y axis"
        )
        parser.add_argument(
            "-r", "--radius", type=int, default=8, help="radius of central lead"
        )
        parser.add_argument(
            "-n", "--Niter", type=int, default=1500, help="number of iterations"
        )

        args = parser.parse_args()

        return args.Nx, args.Ny, args.radius, args.Niter

\end{lstlisting}
\pagebreak

\subsection{Initialize the Potential Array} \label{sec:contour}
The potential array $\phi$ has been generated by:
\begin{lstlisting}
    phi = np.zeros((Ny, Nx))  # allocate the potential array
    X, Y = np.meshgrid(np.linspace(-0.5, 0.5, Nx), np.linspace(-0.5, 0.5, Ny))

    # scaled radius = 0.35 for default args, 5% margin for floating point errors
    ii = np.where(X ** 2 + Y ** 2 <= (1.05 * radius / (min(Nx, Ny) - 1)) ** 2)
    phi[ii] = 1.0  # initialize the potential array
\end{lstlisting}
The \code{contour_plot()} function obtains a contour plot of the potential,
with the nodes in the $V=1$ region marked red, by the following:
\begin{lstlisting}
    plt.clabel(plt.contour(X, Y, phi))
    plt.scatter(X[0, ii[1]], Y[ii[0], 0], color="r", label="$V=1$")
\end{lstlisting}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{3.png}
\end{figure}

\subsection{Perform the Iterations}
The \code{iterate()} function realizes the following logic:
\begin{lstlisting}
    for k in range(Niter)
        save copy of phi
        update phi array
        assert boundaries
        errors[k]=(abs(phi-oldphi)).max();
    #end
\end{lstlisting}
\pagebreak

Through this loop:
\begin{lstlisting}
    for k in range(Niter):
        oldphi = phi.copy()

        # updating the potential
        phi[1:-1, 1:-1] = (  # update interior points to average of neighbors
            phi[1:-1, :-2] + phi[1:-1, 2:] + phi[:-2, 1:-1] + phi[2:, 1:-1]
        ) / 4

        # boundary conditions
        phi[1:-1, 0] = phi[1:-1, 1]  # left
        phi[1:-1, -1] = phi[1:-1, -2]  # right
        phi[-1, :] = phi[-2, :]  # top
        phi[ii] = 1.0  # central area corresponding to electrodes to 1

        errors[k] = np.max(np.abs(phi - oldphi))
\end{lstlisting}
Subsections \ref{sec:updating} and \ref{sec:boundaries} explain the updating of the potential
and the boundary conditions respectively.

\subsection{Updating the Potential} \label{sec:updating}
The potential is updated according to the following equation:
\begin{equation*}
    \phi_{i,j} = \frac{\phi_{i-1,j} + \phi_{i+1,j} + \phi_{i,j-1} + \phi_{i,j+1}}{4}
\end{equation*}
This is accomplished by the following code:
\begin{lstlisting}
    phi[1:-1, 1:-1] = (  # update interior points to average of neighbors
        phi[1:-1, :-2] + phi[1:-1, 2:] + phi[:-2, 1:-1] + phi[2:, 1:-1]
    ) / 4
\end{lstlisting}

\subsection{Boundary Conditions} \label{sec:boundaries}
The boundary condition at the left boundary, for example, is:
\begin{equation*}
    \frac{\partial \phi}{\partial x} = 0
\end{equation*}
This is accomplished by the following code:
\begin{lstlisting}
    phi[1:-1, 0] = phi[1:-1, 1]  # left
    phi[1:-1, -1] = phi[1:-1, -2]  # right
    phi[-1, :] = phi[-2, :]  # top
    phi[ii] = 1.0  # central area corresponding to electrodes to 1
\end{lstlisting}
The potential of the nodes corresponding to the electrode region has been ensured to be $1$.
\pagebreak

\subsection{Graph the Results}
The \code{plot_errors()} function plots the errors vs iteration number:
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{7a.png}
    \caption*{log-log scale}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{7b.png}
    \caption*{semilog scale}
\end{figure}
\pagebreak

To see individual data points, every 50\textsuperscript{th} point has been plotted:
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{7c.png}
\end{figure}

The data may be fit to an exponential, especially for larger iteration numbers (i.e. beyond 500):
\begin{equation*}
    y = Ae^{Bx} \implies \log y = \log A + Bx
\end{equation*}
The code used for this is:
\begin{lstlisting}
    fit1 = lstsq(np.c_[np.ones(len(errors)), n], np.log(errors))[0]
    fit2 = lstsq(np.c_[np.ones(len(errors)), n][500:], np.log(errors)[500:])[0]

    plt.semilogy(np.exp(fit1[0] + fit1[1] * n), "r", lw=3, label="fit1")
    plt.semilogy(np.exp(fit2[0] + fit2[1] * n), "g", label="fit2")
\end{lstlisting}
Where \code{n} is defined as \code{np.arange(len(errors))}.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{7d.png}
\end{figure}

\subsection{Surface Plot of Potential}
The \code{surface_plot()} generates the surface plot by the following code:
\begin{lstlisting}
    ax = plt.figure(figsize=(8, 8)).add_subplot(projection="3d")  # new way
    plt.colorbar(ax.plot_surface(X, Y, phi, rstride=1, cstride=1, cmap=cm.jet))
\end{lstlisting}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.425]{8.png}
\end{figure}

\subsection{Contour Plot of Potential}
The same \code{contour_plot()} function from Subsection \ref{sec:contour} has been used:
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.425]{9.png}
\end{figure}

\subsection{Vector Plot of Currents}
The currents are given by the following equations:
\begin{align*}
    J_{x,ij} & = \frac{1}{2} (\phi_{i,j-1} - \phi_{i,j+1}) \\
    J_{y,ij} & = \frac{1}{2} (\phi_{i-1,j} - \phi_{i+1,j})
\end{align*}

The \code{plot_currents()} function implements these equations as:
\begin{lstlisting}
    Jx, Jy = np.zeros_like(phi), np.zeros_like(phi)
    Jx[:, 1:-1] = (phi[:, :-2] - phi[:, 2:]) / 2
    Jy[1:-1, :] = (phi[:-2, :] - phi[2:, :]) / 2
\end{lstlisting}

These matrices are subsequently plotted as:
\begin{lstlisting}
    plt.quiver(X, Y, Jx, Jy, scale=4)
    plt.scatter(X[0, ii[1]], Y[ii[0], 0], color="r", label="$V=1$")
\end{lstlisting}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{10.png}
\end{figure}

\section{Conclusion}
The Laplace Equation has been solved using discrete differentiation iteratively.
This method of solving the Laplace Equation is generally not recommended.
This is because of the very slow coefficient with which the error reduces.

\end{document}


\documentclass{article}
\usepackage{graphicx}
\usepackage{amssymb,amsmath}
\usepackage{fancyhdr}
\pagestyle{fancyplain}
\rhead{Prepared by Adam Beardsley on \today}

\begin{document}

\section{Weights and variance of gridded visibilities}
We start with the rms of a visibility,
\begin{equation}\label{eq:vis_rms}
\sigma_v = \frac{2 k_B T_{sys}}{A_e \sqrt{d\!f \tau}}
\end{equation}
where $k_B$ is Boltzmann's constant, $T_{sys}$ is the system temperature, $A_e$ is the effective collecting area, $df$ is the frequency channel width, and $\tau$ is the single snapshot observation time.  We can easily plug in true observational parameters to give us the rms for individual visibilities.

\begin{equation}
\sigma_v = 2761.3006 \left(\frac{T_{sys}}{\text{K}}\right) \left(\frac{\text{m}^2}{A_e}\right) \left(\frac{\text{Hz}}{df}\right)^{1/2} \left(\frac{\text{sec}}{\tau}\right)^{1/2} \text{Jy}
\end{equation}
For example, if we have $T_{sys} = 440$ K, $A_e = 16$ m$^2$, $df = 40$ kHz, and $\tau = 5$ seconds, I get $\sigma_v \approx 169.8$ Jy.

Ian uses the model beam to grid the data (or the weights) into a regular $uv$ grid. As an example, let's look at a simplified case of two overlapping visibilities, and consider the gridded value and variance of a pixel in the overlapping region.
\begin{figure}[h!]
\includegraphics[width=\columnwidth]{vis_grid.png}
\caption{Gridding two visibilities}
\end{figure}

Each visibility ($v_1$ and $v_2$)is multiplied by the beam value at the pixel in question ($g_1$ and $g_2$) and added to produce the gridded visibility.
\begin{equation}
v_g = g_1 v_1 + g_2 v_2
\end{equation}
The variance on $v_g$ can easily be calculated using error propagation.
\begin{equation}\label{eq:grid_var}
\sigma_g = \sqrt{\left(g_1\sigma_1\right)^2 + \left(g_2 \sigma_2\right)^2} \approx \sigma_v \sqrt{g_1^2 + g_2^2}
\end{equation}
In the case that our antennas are identical, all the values in Eq. \ref{eq:vis_rms} are identical, so we can replace $\sigma_1$ and $\sigma_2$ with a common $\sigma_v$.

\vspace{.2cm}
[Adding snapshots in healpix\ldots need to figure this out\ldots]
\vspace{.2cm}

Ignoring the healpix business for now, we want to know what happens to our noise when we divide by the weights in $(u,v,f)$ space. The ``weights" we use are the sum of beam models in each pixel. In our simple example,
\begin{equation}
w_g = g_1 + g_2.
\end{equation}
This weight is an accurate description of how well we sampled each pixel on the $uv$ plane. Our best estimate of the apparent sky is then
\begin{equation}
\hat{v} = \frac{v_g}{w_g} = \frac{g_1v_1+g_2v_2}{g_1+g_2}.
\end{equation}

Again we can simply propagate error.
\begin{equation}
\sigma_{\hat{v}} = \frac{\sigma_g}{w_g} = \sigma_v \frac{\sqrt{g_1^2 + g_2^2}}{g_1 + g_2}
\end{equation}

From here we can form our covariance matrix and propagate through the frequency Fourier transform and square to power spectrum as Bryna has been doing.

{\bf Conclusion:} \, To me it seems like we need both the weights cube ($g_1 + g_2$ in the example above) as well as a cube where the beams were squared before the gridding ($g_1^2 + g_2^2$) in order to propagate the rms correctly.

\section{Covariance between grid points}
Because the same data is being gridded to multiple grid points, this gridded data is highly covariant. Here I will quantize this covariance.

Consider two grid points, indexed with $i$ and $j$, with $n$ visibilities contributing. In general the visibilities contributing to both grid points do not need to be identical (we are interested in cases where there is some overlap), but let $n$ represent the union of the two sets of visibilities. Then we have the gridded visibilities,
\begin{subequations} \label{eq:two_grids}
\begin{align}
v_i &= \sum_n g_{n,i} v_n  \\
v_j &= \sum_n g_{n,j} v_n
\end{align}
\end{subequations}
We wish to calculate the covariance.
\begin{equation}
C_{ij} = \left<(v_i-\bar{v}_i)(v_j-\bar{v}_j)^*\right>
\end{equation}
Angled brackets ($<...>$) represent an expectation value, and a bar is shorthand for the same. We can simplify this equation slightly before proceeding.
\begin{subequations}
\begin{align}
C_{ij} &= \left<v_iv_j^* - v_i\bar{v}_j^* - \bar{v}_iv_j^* + \bar{v}_i\bar{v}_j^*\right> \\
&= \left<v_iv_j^*\right> - \bar{v}_i\bar{v}_j^* - \bar{v}_i\bar{v}_j^* + \bar{v}_i\bar{v}_j^* \\
& = \left<v_iv_j^*\right> - \bar{v}_i\bar{v}_j^*
\end{align}
\end{subequations}

Now we plug in Eq. \ref{eq:two_grids}, and group terms appropriately.
\begin{subequations}
\begin{align}
C_{ij} &= \left<\sum_n g_{n,i}v_n \sum_m g_{m,j}^* v_m^* \right> - \sum_n g_{n,i}\bar{v}_n \sum_m g_{m,j}^* \bar{v}_m^* \\
&= \sum_n\sum_m\left(g_{n,i}g_{m,j}^* \left<v_n v_m^*\right> - g_{n,i}g_{m,j}^*\bar{v}_n\bar{v}_m^*   \right) \\
&= \sum_n\sum_m g_{n,i}g_{m,j}^* C_{nm}
\end{align}
\end{subequations}

We have now expressed our gridded covariance matrix in terms of the raw visibility covariance matrix. Thus far we have made no assumptions about the covariance of the visibilities, however, in order to simplify further I will assume that this covariance matrix is diagonal and all visibilities have the same variance (as we assumed above). Then we have the result,
\begin{equation}
C_{ij} =  \sigma_v^2 \sum_n g_{n,i}g_{n,j}^*.
\end{equation}
Now the sum over $n$ can be restricted to the intersection of visibilities that contribute to \emph{both} grid points. It is nice to see that in the case $i=j$, this result conforms to Eq. \ref{eq:grid_var}. Additionally, this is exactly the Holographic Mapping function used in FHD, which Ian already computes for every snapshot.

{\bf Conclusion:} The covariance matrix between grid points for a single snapshot is given simply by the Holographic Mapping function used in FHD. In principle we should be able to transform this function to the image space, add in healpix, and transform back in the same way we treat the data. The difficulty is the size of this matrix. It is sparse, which should help greatly in the transformation to image space, but it is not sparse there, and the transform back may be difficult. In addition, this adds to the work needed for the Fourier transform over frequency when calculating power spectra.

\section{Error propagation to power spectrum}

In the next steps we wish to arrive at a power spectrum measurement, using two interleaved data cubes along with their variances. I will assume we performed the frequency Fourier transform on the data and propagated the covariance cubes by applying the Fourier transform matrix on both sides. In this space, we have cubes $a_1(u,v,\eta)$ and $a_2(u,v,\eta)$ where 1 and 2 correspond to the two interleaved cubes. We also have $b_1$ and $b_2$ for the orthogonal mode (e.g. sine and cosine), but they are separated in most of the math, so I will focus on $a$'s for now.

The first thing we need to do is add and subtract the two cubes, accounting for the weights. From here on out, I will assume the ``weights" are the inverse of the variance, rather than the gridded constants from the previous section.
\begin{equation}
a_{sum} = \frac{w_{a1}a_1 +w_{a2} a_2}{w_{a1}+w_{a2}} ; \; \; \;
a_{sub} = \frac{w_{a1}a_1 -w_{a2} a_2}{w_{a1}+w_{a2}}
\end{equation}
The variance of the results are
\begin{equation}
\sigma^2_{sum} = \sigma^2_{sub} = \frac{1}{w_{a1} + w_{a2}}.
\end{equation}
Because all we have done thus far is scale by weights and add, I believe we can perform this step in the $(u,v,f)$ space due to the linearity of the Fourier transform. This should allow us to avoid translating two weights cubes to $(u,v,\eta)$ space.

The next step is to perform a magnitude squared on both $a_{sum}$ and $a_{sub}$. The math is a little messy, so I will demonstrate it for an arbitrary complex number drawn from a symmetric gaussian distribution. We start with a probability distribution function for a number $c$.
\begin{equation}
\mathrm{Pr}(c) = \frac{1}{\pi \sigma^2} e^{-\left|c-\mu\right|^2/\sigma^2}
\end{equation}
Here $\mu$ is the mean complex value of the distribution, and $\sigma$ is the standard deviation of the \emph{complex number} $c$. It is important to distinguish this $\sigma$ from the standard deviation of the real and imaginary parts separately (which would be a factor of $\sqrt{2}$ smaller). Also note that the visibility rms given at the beginning of the last section is actually the real/imaginary version. So things aren't exactly perfect from here...

Next we find the pdf of $|c|$ which is found by integrating a ring of constant magnitude,
\begin{equation}
\mathrm{Pr}(|c|) = \int_0^{2\pi}d\phi \mathrm{Pr}(c)|c| = \frac{2 |c|}{\sigma^2} e^{-(|c|^2+|\mu|^2)/\sigma^2} I_0(\frac{2|c||\mu|}{\sigma^2})
\end{equation}
where $I_0$ is the $0^{\mathrm{th}}$ modified Bessel function. A change of variables takes us to the pdf for $|c|^2$.
\begin{equation}
\mathrm{Pr}(|c|^2) = \frac{d |c|}{d |c|^2} \mathrm{Pr}(|c|) = \frac{1}{\sigma^2} e^{-(|c|^2+|\mu|^2)/\sigma^2} I_0(\frac{2|c||\mu|}{\sigma^2})
\end{equation}

The stats we care most about here are the expectation value of $|c|^2$ and the variance of $|c|^2$. They are
\begin{subequations}
\begin{align}
\left< |c|^2 \right> & = |\mu|^2 + \sigma^2\\
\mathrm{var}(|c|^2) & = \sigma^2\left(2 |\mu|^2 + \sigma^2\right).
\end{align}
\end{subequations}

Back to the sum and difference cubes, we can now determine the expected values and variances of the magnitudes squared (taking $S$ to be the signal, and $\sigma^2$ is the inverse of the respective weight).
\begin{subequations}
\begin{align}
\left<\left|a_1+a_2\right|^2\right> & = |S|^2 + \sigma_{sum}^2 \\
\left<\left|a_1-a_2\right|^2\right> & = \sigma_{sub}^2
\end{align}
\end{subequations}
\begin{subequations}
\begin{align}
\mathrm{var}\left(\left|a_1+a_2\right|^2\right) & = \sigma_{sum}^2 \left(2 |S|^2 + \sigma_{sum}^2\right) \\
\mathrm{var}\left(\left|a_1-a_2\right|^2\right) & = \sigma_{sub}^4
\end{align}
\end{subequations}

We are almost there. Because our sum term has noise in its expected value, we must subtract the pure noise term to arrive at a best estimate of the signal.
\begin{equation}
\left|\hat{S}_a\right|^2 = \left|a_1+a_2\right|^2 - \left|a_1-a_2\right|^2
\end{equation}
Note that even though the two terms can have very different variances, this should \emph{not} be a weighted sum, or the noise would not subtract properly. The variance is
\begin{equation}
\sigma^2_{S,a} = \mathrm{var}\left(\left|a_1+a_2\right|^2\right) + \mathrm{var}\left(\left|a_1-a_2\right|^2\right) = 2 \sigma_{sum}^2\left(|S|^2 + \sigma_{sum}^2\right)
\end{equation}
where I have used the fact that $\sigma_{sum}=\sigma_{sub}$ to simplify slightly. At first look I thought these terms would be covariant and the error propagation would not be so easy. However, simple simulations have shown that the covariant terms are negligible. I have not fully understood this, but I think it has to do with the particular linear combinations being independent (similar to changing bases in vector space).

Finally we can include the $b$ terms to help our estimate slightly.
\begin{equation}
\left|\hat{S}\right|^2 = \frac{w_{S,a}\left|\hat{S}_a\right|^2 + w_{S,b}\left|\hat{S}_b\right|^2}{w_{S,a}+w_{S,b}}
\end{equation}
With a variance on our final estimate being
\begin{equation}
\sigma^2_S = \frac{1}{w_{S,a}+w_{S,b}}.
\end{equation}

\section{PDF after averaging bins}

In this section I will describe the probability distribution function when we square complex numbers, then average bin (rings for 2D PS, shells for 1D PS). As in the last section, I will treat the two orthogonal modes in a $k$ bin separately (e.g. sines and cosines). They are combined in the averaging step with all the other modes in the bin.

We start with the PDF of a symmetric complex gaussian distributed number. Some of this is repeated from above, but I wish to put everything together. To conform notation to what Bryna uses in her code, $\sigma$ will now represent the uncertainty on the real and imaginary parts individually. I will also drop the mean value because we are currently particularly interested in the noise cubes.
\begin{equation}
\mathrm{Pr}(c) = \frac{1}{2\pi \sigma^2}e^{-|c|^2/2\sigma^2}
\end{equation}

Taking magnitude squared of $c$ gives us an exponential distribution with parameter $\lambda=1/2\sigma^2$.
\begin{equation}
\mathrm{Pr}(|c|^2) = \frac{1}{2\sigma^2}e^{-|c|^2/2\sigma^2}
\end{equation}
The expectation value and variance of this distribution are $1/\lambda = 2\sigma^2$ and $1/\lambda^2 = 4\sigma^4$ respectively.

The next step is to perform a weighted average in either rings or spherical shells. We assign the above distribution for each voxel $i$ and weight by the inverse variance to sum. Here we include the sine and cosine modes in the sum.
\begin{equation}
\overline{|c|^2} = \frac{\sum\lambda_i^2 |c_i|^2}{\sum \lambda_i^2}
\end{equation}
The expectation value and variance of the weighted sum are given by
\begin{subequations}
\begin{align}
\left<\overline{|c|^2}\right> & = \frac{\sum\lambda_i^2 \left<|c_i|\right>}{\sum \lambda_i^2} = \frac{\sum\lambda_i}{\sum \lambda_i^2} \\
\mathrm{var}(\overline{|c|^2})& = \frac{1}{\sum \lambda_i^2}.
\end{align}
\end{subequations}

The sum of exponentially distributed numbers with equal parameters yields an Erlang distributed number. The weighted sum does not exactly yield an Erlang distribution, but if the weights are reasonably uniform (within an order of magnitude of one another), we can well approximate the resulting distribution as Erlang with effective parameters. The PDF of an Erlang is given by
\begin{equation}\label{eq:erlang}
\mathrm{Pr}(z) = \frac{z^{n-1} e^{-z/\beta}}{\beta^n \Gamma(n)}
\end{equation}
where $n$ would be the number of voxels that went into the average if all weights were the same, and $\beta$ would be $1/\lambda$. Because our weights are not all the same some voxels contribute more than others, which leads to an effective number that went into the average less than the true number, and a variance more heavily weighted by the small variance voxels. We can solve for the effective parameters by writing down the mean and variance of this distribution, and setting them equal to those for our weighted average.
\begin{subequations}
\begin{align}
\left<z \right> & = n_{\mathrm{eff}} \beta_{\mathrm{eff}} = \frac{\sum\lambda_i}{\sum \lambda_i^2} \\
\mathrm{var}(z) & = n_{\mathrm{eff}} \beta_{\mathrm{eff}}^2 = \frac{1}{\sum \lambda_i^2}
\end{align}
\end{subequations}
Solving\ldots
\begin{subequations}\label{eq:erlang_params}
\begin{align}
n_{\mathrm{eff}} & = \frac{\left(\sum\lambda_i\right)^2}{\sum \lambda_i^2} \\
\beta_{\mathrm{eff}} & = \frac{1}{\sum\lambda_i}
\end{align}
\end{subequations}

We can test this parameterized fit with some simulated data. In Figure \ref{fig:erlang} we plot a histogram of simulated data obtained from squaring and averaging complex random numbers (ie. starting at the beginning of this section).

The fit accurately predicts the shape of the distribution for a few voxels as well as for many. Where the fit breaks down is when the weights that go into the average are allowed to vary significantly. In the figure titles, the quantity `sigma SNR' refers to the average sigma divided by the standard deviation of the sigmas that went into the original simulation. As this number decreases, the predicted distribution (red line) deviates from the simulated data. In particular, we see int he last plot, when the sigma SNR is about 3 that the predicted distribution is not as peaked as the simulation.

More investigation is needed to determine if the voxels going into our averages will satisfy the large sigma SNR condition. However, this method can be tentatively used to predict the noise power and uncertainty in EoR data.

\begin{figure}
\includegraphics[width=\columnwidth]{erlang.pdf}
\caption{Simulated noise data against fit Erlang distributions}
\label{fig:erlang}
\end{figure}

\end{document}


% =============================================================================

\subsection{General}

\begin{itemize}

\item $x \ASN     y$
      is used to denote an assignment of the value $y$
      to the variable $x$.
\item $x \RAND    Y$
      is used to denote an assignment of a   value $y$,
      sampled uniformly at from the set $Y$ st. $y \in Y$,
      to the variable $x$.
\item $x \TEST{=} y$
      is used to denote a  test or comparison,
      in this case equality,
      between $x$ and $y$ that yields a Boolean (i.e., \TRUE or \FALSE) result.
\item $x \RANGE   y$
      denotes the range of values between $x$ and $y$ inclusive; this implies
      a need to generate intermediate values, st. the notation is only useful
      iff. doing so is obvious {\em and} unambiguous.

\item  $\CARD{X}$
      denotes the cardinality (or size) of some object $X$.
\item $\INDEX{X}{i}$
      refers to an indexed (or numbered)
      element   within some object     $X$, namely the $i$-th such element.
\item $\FIELD{X}{Y}$
      refers to a  named (or labelled)
      field $Y$ within some object     $X$.
\item $\SCOPE{X}{Y}$
      refers to a  named (or labelled)
      field $Y$ within some name-space $X$;
      this is useful to disambiguate different $Y$ with the same identifier.

\end{itemize}

% -----------------------------------------------------------------------------

\subsection{Numbers}

\begin{itemize}

\item $\RADIX{x}{b}$ denotes $x$ is expressed in radix- or base-$b$; where no
      base is specified, it is safe to assume decimal (i.e., that $b = 10$).
\item $\CARD{\RADIX{x}{b}}$ denotes the number of digits in $x$, if expressed
      in base-$b$.
\item Unless otherwise stated, a little-endian digit ordering is assumed; the
      least-significant (resp. most-significant) digit is thus $\INDEX{x}{0}$
      (resp. $\INDEX{x}{n}$).

\end{itemize}

% -----------------------------------------------------------------------------

\subsection{Bits}

\begin{itemize}

\item The following terms refer to fixed-length (sub-)sequences of $n$ bits
      \[
      \begin{array}{lcc@{\;}c@{\;}r}
      \mbox      {crumb} &\Longrightarrow& n &=&  2 \\
      \mbox     {nibble} &\Longrightarrow& n &=&  4 \\
      \mbox       {byte} &\Longrightarrow& n &=&  8 \\
      \mbox  {half-word} &\Longrightarrow& n &=& 16 \\
      \mbox       {word} &\Longrightarrow& n &=& 32 \\
      \mbox{double-word} &\Longrightarrow& n &=& 64 \\
      \end{array}
      \]
      where perhaps all but the first are fairly standard.
\item Where appropriate, a sequence $x$ of $n$ bits can be interpreted as an
      element of the polynomial ring $\B{F}_2[\IND{x}]$
      (i.e. a binary polynomial);
      the $i$-bit of $x$ basically represents the $i$-th coefficient, st.
      $
      x \mapsto x( \IND{x} )
      $
      because
      $
      x_i \mapsto x_i \cdot \IND{x}^i .
      $
\item The operators $\NOT$, $\AND$, $\IOR$, and $\XOR$ denote Boolean NOT, AND,
      OR, and XOR respectively, with $\NAND$, $\NIOR$ and $\NXOR$ denoting NAND,
      NOR, and NXOR; all of these may be overloaded to cater for bit-sequences
      rather than simply bits.
\item $\HW( X )$
      denotes the Hamming weight   of      bit-sequence  $X$,         st.
      \[
      \HW( X )    = \sum_{i=0}^{i<n} \INDEX{X}{i}                   .
      \]
\item $\HD( X )$
      denotes the Hamming distance between bit-sequences $X$ and $Y$, st.
      \[
      \HD( X, Y ) = \sum_{i=0}^{i<n} \INDEX{X}{i} \XOR \INDEX{Y}{i} .
      \]
\item Given a bit-sequence
      \[
      X = \LIST{ \INDEX{X}{n-1}, \ldots, \INDEX{X}{1}, \INDEX{X}{0} }
      \]
      and assuming $w$ divides $\CARD{X}$, $\INDEX[w]{X}{i}$ is used to denote
      the $i$-th sub-sequence of length $w$ in $X$: for $n = 16$, for example,
      we have that
      \[
      \begin{array}{lcl}
      \INDEX[4]{X}{0} &=& \LIST{ \INDEX{X}{ 3}, \INDEX{X}{ 2}, \INDEX{X}{ 1}, \INDEX{X}{ 0}                                                             } \\
      \INDEX[4]{X}{2} &=& \LIST{ \INDEX{X}{11}, \INDEX{X}{10}, \INDEX{X}{ 9}, \INDEX{X}{ 8}                                                             } \\
      \INDEX[8]{X}{0} &=& \LIST{ \INDEX{X}{ 7}, \INDEX{X}{ 6}, \INDEX{X}{ 5}, \INDEX{X}{ 4}, \INDEX{X}{ 3}, \INDEX{X}{ 2}, \INDEX{X}{ 1}, \INDEX{X}{ 0} } \\
      \INDEX[8]{X}{1} &=& \LIST{ \INDEX{X}{15}, \INDEX{X}{14}, \INDEX{X}{13}, \INDEX{X}{12}, \INDEX{X}{11}, \INDEX{X}{10}, \INDEX{X}{ 9}, \INDEX{X}{ 8} } \\
      \end{array}
      \]
      for example.  In essence, this is a short-hand st.
      \[
      \INDEX[w]{X}{i} \equiv \LIST{ \INDEX[w]{X}{w \cdot i}, \INDEX[w]{X}{w \cdot i + 1}, \ldots, \INDEX[w]{X}{w \cdot i + w - 1} } .
      \]
\item $\LSB[l]{X}$ (resp. $\MSB[l]{X}$) denotes the $l$ least- (resp. most-)
      significant bits of $X$ (where we assume $l = 1$ if omitted): given a
      bit-sequence
      \[
      X = \LIST{ \INDEX{X}{n-1}, \ldots, \INDEX{X}{1}, \INDEX{X}{0} } ,
      \]
      for example, and assuming $l \leq \CARD{X}$, we have that
      \[
      \begin{array}{lcl}
      \LSB[l]{X} &=& \LIST{ \INDEX{X}{l-1},                 \ldots, \INDEX{X}{  1}, \INDEX{X}{  0} } \\
      \MSB[l]{X} &=& \LIST{ \INDEX{X}{n-1}, \INDEX{X}{n-2}, \ldots                  \INDEX{X}{n-l} } \\
      \end{array}
      \]

\end{itemize}

% -----------------------------------------------------------------------------

\subsection{Operations}

\begin{itemize}

\item The operators $\LSH$ and $\RSH$ denote left- and right-shift; $\LRT$ and
      $\RRT$ denote left- and right-rotate (beware of context: $\ll$ and $\gg$
      are also used to denote ``much less than'' and ``much greater than'').
\item The operators
      $\oplus$
      and
      $\otimes$
      respectively denote addition and multiplication in $\B{F}_2[\IND{x}]$.
\item For some $x$,
      we let
      $\EXT[w]{0  }( x )$
      and
      $\EXT[w]{\pm}( x )$
      respectively denote
      zero- or sign-extension to $w$-bits
      (allowing omission of either specifier where appropriate).
\item For some operator $\odot$,
      we let
      $\OP[w][s]{\odot}$
      and
      $\OP[w][u]{\odot}$
      respectively denote
      $w$-bit signed and unsigned variants
      (allowing omission of either specifier where appropriate).

\end{itemize}

% -----------------------------------------------------------------------------

\subsection{Cryptography}

\begin{itemize}

\item $\ROUND{x}{k}$ denotes some quantity $x$ relating to the $k$-th
      round (e.g., the $k$-th round key).
\item The AES round functions are denoted using
      $\SCOPE{\ID{AES}}{\ALG{Key-Addition}}$,
      $\SCOPE{\ID{AES}}{\ALG{Sub-Bytes}}$,
      $\SCOPE{\ID{AES}}{\ALG{Shift-Rows}}$,
      and
      $\SCOPE{\ID{AES}}{\ALG{Mix-Columns}}$;
      note that
      a) $\SCOPE{\ID{AES}}{\ALG{Mix-Column}}$ is applied per column of the
         state matrix to realise $\SCOPE{\ID{AES}}{\ALG{Mix-Columns}}$, so
         the former is basically a $1$-column version of the latter,
      b) $\SCOPE{\ID{AES}}{\ALG{S-Box}}$ denotes the AES S-box,
         and
      c) $\SCOPE{\ID{AES}}{\ALG{X}}^{-1}$ denotes the inverse of $\ALG{X}$,
         e.g., $\SCOPE{\ID{AES}}{\ALG{S-Box}}^{-1}$ is the inverse S-box.

\end{itemize}

% -----------------------------------------------------------------------------

\subsection{Architecture}

\begin{itemize}

\item $\RNG$
      denotes the Random Number Generator (RNG) object;
      we equip this object with three operations, namely
      \[
      \begin{array}{c@{\;}c@{\;}ccl}
      x    &\TEST{\ASN}& \RNG &:& \mbox{test                    the RNG state} \\
      x    &      \ASN & \RNG &:& \mbox{sample entropy $x$ from the RNG      } \\
      \RNG &      \ASN & x    &:& \mbox{inject entropy $x$ into the RNG      } \\
      \end{array}
      \]
\item $\MEM$
      denotes the byte-addressable memory;
      $\MEM[*][i]$
      denotes the $i$-th,
            $8$-bit
      entry in said memory.
\item $\GPR$
      denotes the
      general-purpose register file;
      $\GPR[*][i]$
      denotes the $i$-th,
      $\RVXLEN$-bit
      entry in said register file.

\end{itemize}

% =============================================================================


\section{Track}

\begin{itemize}
\item[\texttt{(M1.1)}] is a \emph{Support Vector Classifier (SVC)} with the \emph{hinge} loss.

\begin{itemize}
\item[\texttt{(A1.1.1)}] is a \emph{momentum descent} approach~\cite{polyak1964some, nesterov1998introductory}, an \emph{accelerated gradient} method for solving the SVC in its \emph{primal} formulation.

\item[\texttt{(A1.1.2)}] is the \emph{Sequential Minimal Optimization (SMO)} algorithm~\cite{platt1998sequential, keerthi2001improvements}, an ad hoc \emph{active set} method for training a SVC in its \emph{Wolfe dual} formulation with \emph{linear}, \emph{polynomial} and \emph{gaussian} kernels.

\item[\texttt{(A1.1.3)}] is the \emph{AdaGrad} algorithm~\cite{duchi2011adaptive}, a \emph{deflected subgradient} method for solving the SVC in its \emph{Lagrangian dual} formulation with \emph{linear}, \emph{polynomial} and \emph{gaussian} kernels.
\end{itemize}

\end{itemize}

\begin{itemize}
\item[\texttt{(M1.2)}] is a \emph{Support Vector Classifier (SVC)} with the \emph{squared hinge} loss.

\begin{itemize}
\item[\texttt{(A1.2.1)}] is a \emph{momentum descent} approach~\cite{polyak1964some, nesterov1998introductory}, an \emph{accelerated gradient} method for solving the SVC in its \emph{primal} formulation.
\end{itemize}

\end{itemize}

\bigskip

\begin{itemize}
\item[\texttt{(M2.1)}] is a \emph{Support Vector Regression (SVR)} with the \emph{epsilon-insensitive} loss.

\begin{itemize}
\item[\texttt{(A2.1.1)}] is a \emph{momentum descent} approach~\cite{polyak1964some, nesterov1998introductory}, an \emph{accelerated gradient} method for solving the SVR in its \emph{primal} formulation.

\item[\texttt{(A2.1.2)}] is the \emph{Sequential Minimal Optimization (SMO)} algorithm~\cite{flake2002efficient, shevade1999improvements}, an ad hoc \emph{active set} method for training a SVR in its \emph{Wolfe dual} formulation with \emph{linear}, \emph{polynomial} and \emph{gaussian} kernels.

\item[\texttt{(A2.1.3)}] is the \emph{AdaGrad} algorithm~\cite{duchi2011adaptive}, a \emph{deflected subgradient} method for solving the SVR in its \emph{Lagrangian dual} formulation with \emph{linear}, \emph{polynomial} and \emph{gaussian} kernels.
\end{itemize}

\end{itemize}

\begin{itemize}
\item[\texttt{(M2.2)}] is a \emph{Support Vector Regression (SVR)} with the \emph{squared epsilon-insensitive} loss.

\begin{itemize}
\item[\texttt{(A2.2.1)}] is a \emph{momentum descent} approach~\cite{polyak1964some, nesterov1998introductory}, an \emph{accelerated gradient} method for solving the SVR in its \emph{primal} formulation.
\end{itemize}

\end{itemize}