\chapter{Random Forests \label{chapter:randomforests}}

In Chapter~\ref{chapter:decisiontrees}, we saw how individual decision trees are learned from training data. These days, decision trees are mainly used as parts of \textbf{ensembles}, collections of models whose predictions are combined to produce a final answer.

A \textbf{random forest} is an ensemble of decision trees whose predictions are [mostly] uncorrelated. Each tree is built using a subset of the training data and a subset of the features. The trees' predictions are then combined using a voting or weighting scheme. The same basic methodology works for several different supervised learning problems, including classification (Chapter~\ref{chapter:classification}), regression (Chapter~\ref{chapter:regression}), and survival analysis (Chapter~\ref{chapter:km}). A key advantage of random forests is that they are non-parametric, meaning that they make no distributional or functional assumptions about the relationships between the predictors and the outcome. Another advantage is that the fitting of individual trees happens independently and can be \textbf{parallelized}.

\vspace{4mm}

\begin{question}{}
If random forests are so great, why are linear, logistic, and Cox proportional hazards regression models still the standard approaches to predicting continuous, categorical, and survival outcomes in clinical research? What do you think are some of the main drawbacks of random forests and other ensemble methods?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Building a Random Forest}

Assume we have a training dataset of $N$ samples and $P$ predictors. The basic strategy for building a random forest is as follows\footnote{See \emph{Elements of Statistical Learning}, Chapter 15, Algorithm 15.1.}:

\begin{enumerate}
\item For $b = 1, \dots, B$, where $B$ is the desired number of trees:
  \begin{enumerate}
  \item Draw a bootstrap sample of size $n$ from the training data.
  \item Grow a tree, $T_b$, on the bootstrap sample by recursively repeating the following steps for each terminal node of the tree until the minimum node size, $n_\text{min}$, is reached:
    \begin{enumerate}
    \item Select $p$ predictors at random.
    \item Pick the best predictor and split point.
    \item Split the node into two child nodes.
    \end{enumerate}
  \end{enumerate}
\item Output the ensemble of trees, $T_1, \dots, T_B$.
\end{enumerate}

\vspace{2mm}

\begin{question}{}
This algorithm leaves us with many choices. We must choose $B$, the number of trees; $n$, the size of each bootstrap sample; $p$, the number of predictors evaluated for each split; and $n_\text{min}$, the minimum node size. What impact does each parameter choice have on the properties of the forest, both in terms of the individual trees and the forest as a whole?
\end{question}

\begin{question}{}
What does the ``best'' predictor and split point refer to? How are these choices made for classification and regression models? Think back to our discussion in Chapter~\ref{chapter:decisiontrees}.
\end{question}

The term \textbf{bootstrapping} refers to random sampling with replacement. To create a bootstrap sample of size $n$ from a larger dataset of size $N$, we simply select $n$ items from among the $N$, one at a time, being careful to put each item back between selections. Because we are sampling with replacement, a bootstrap sample will likely contain repeats; this is fine and expected.

The process of averaging model predictions built on different bootstrap samples is called bootstrap aggregating, or \textbf{bagging}. We will learn more about why bagging works in Chapter~\ref{chapter:biasvariance}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Classification Example: Breast Cancer Diagnosis}

Let's revisit the classification example for which we built a single decision tree in Chapter~\ref{chapter:decisiontrees}. The Wisconsin Breast Cancer Dataset contains information about $30$ different imaging features of fine needle aspirate (FNA) samples from breast masses in $569$ study participants. Here are tabular representations of two trees from a 100-tree random forest built on this dataset:

\begin{center}
{\scriptsize \tt
\begin{tabular}{llllrl}
  \toprule
 node\_id & left\_child & right\_child & split\_variable & split\_point & prediction \\
  \midrule
1 & 2 & 3 & area\_mean & 694.10 &  \\
  2 & 4 & 5 & symmetry\_worst & 0.37 &  \\
  3 & 6 & 7 & texture\_mean & 14.09 &  \\
  4 & 8 & 9 & concave.points\_mean & 0.05 &  \\
  5 & 10 & 11 & radius\_worst & 14.84 &  \\
  6 & 12 & 13 & compactness\_se & 0.03 &  \\
  7 & 14 & 15 & concave.points\_mean & 0.05 &  \\
  8 & 16 & 17 & area\_se & 42.19 &  \\
  9 & 18 & 19 & smoothness\_worst & 0.13 &  \\
  10 & 0 & 0 &  & 0.00 & B \\
  11 & 0 & 0 &  & 0.00 & M \\
  12 & 0 & 0 &  & 0.00 & B \\
  13 & 0 & 0 &  & 0.00 & M \\
  14 & 0 & 0 &  & 0.00 & M \\
  15 & 0 & 0 &  & 0.00 & M \\
  16 & 0 & 0 &  & 0.00 & B \\
  17 & 0 & 0 &  & 0.00 & M \\
  18 & 0 & 0 &  & 0.00 & B \\
  19 & 0 & 0 &  & 0.00 & M \\
   \bottomrule
\end{tabular}

\begin{tabular}{llllrl}
  \hline
 node\_id & left\_child & right\_child & split\_variable & split\_point & prediction \\
  \hline
1 & 2 & 3 & perimeter\_worst & 106.10 &  \\
  2 & 4 & 5 & radius\_se & 0.63 &  \\
  3 & 6 & 7 & radius\_mean & 15.04 &  \\
  4 & 8 & 9 & compactness\_worst & 0.76 &  \\
  5 & 10 & 11 & smoothness\_se & 0.01 &  \\
  6 & 12 & 13 & smoothness\_mean & 0.09 &  \\
  7 & 14 & 15 & radius\_worst & 18.23 &  \\
  8 & 16 & 17 & concave.points\_worst & 0.18 &  \\
  9 & 0 & 0 &  & 0.00 & M \\
  10 & 18 & 19 & compactness\_se & 0.01 &  \\
  11 & 0 & 0 &  & 0.00 & B \\
  12 & 0 & 0 &  & 0.00 & B \\
  13 & 0 & 0 &  & 0.00 & M \\
  14 & 0 & 0 &  & 0.00 & M \\
  15 & 0 & 0 &  & 0.00 & M \\
  16 & 0 & 0 &  & 0.00 & B \\
  17 & 0 & 0 &  & 0.00 & M \\
  18 & 0 & 0 &  & 0.00 & M \\
  19 & 0 & 0 &  & 0.00 & B \\
   \hline
\end{tabular}
}
\end{center}

\vspace{3mm}

\begin{question}{}
Draw the two classification trees from the Wisconsin Breast Cancer Dataset random forest that are represented by these tables. Note: if a variable's value is less than the split point at a particular node, the training sample goes to the left.
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Regression Example: Insurance Costs \label{sect:reginsurance}}

The following dataset comes from the book \emph{Machine Learning with R}, by Brett Lantz. It's unclear whether it is real or simulated, but it provides insurance cost information on $1338$ subjects, as well as information about the following predictors:

{\small
\begin{enumerate}[label=(\alph*)]
\item age (age of primary beneficiary)
\item sex (sex of primary beneficiary, labeled ``female'' or ``male'')
\item bmi (body mass index of beneficiary)
\item children (number of children/dependents covered by beneficiary's health insurance)
\item smoker (smoking status of beneficiary)
\item region (the beneficiary's residential area in the U.S.: northeast, southeast, southwest, northwest)
\end{enumerate}
}

The variable \emph{charges} is the outcome of interest; it is the total individual medical costs (in thousands of dollars) billed by the beneficiary's health insurance. Here are tabular representations of two trees from a $100$-tree random forest built on this dataset:
\vspace{-5mm}

\begin{center}
{\scriptsize \tt
\begin{tabular}{llllrl}
  \hline
 node\_id & left\_child & right\_child & split\_variable & split\_point & prediction \\
  \hline
  1 & 2 & 3 & smoker & yes & 13.19 \\
  2 & 4 & 5 & region & NE & 8.65 \\
  3 & 6 & 7 & region & NE, SE, SW & 31.94 \\
  4 & 8 & 9 & children & 2.50 & 7.85 \\
  5 & 10 & 11 & children & 0.50 & 8.92 \\
  6 & 12 & 13 & bmi & 30.01 & 29.21 \\
  7 & 14 & 15 & bmi & 30.30 & 36.26 \\
  8 & 16 & 17 & bmi & 25.19 & 7.29 \\
  9 & 18 & 19 & children & 3.50 & 11.61 \\
  10 & 0 & 0 &  & 0.00 & 8.11 \\
  11 & 0 & 0 &  & 0.00 & 9.55 \\
  12 & 0 & 0 &  & 0.00 & 20.72 \\
  13 & 0 & 0 &  & 0.00 & 41.93 \\
  14 & 0 & 0 &  & 0.00 & 23.15 \\
  15 & 0 & 0 &  & 0.00 & 44.51 \\
  16 & 0 & 0 &  & 0.00 & 10.95 \\
  17 & 0 & 0 &  & 0.00 & 6.82 \\
  18 & 0 & 0 &  & 0.00 & 12.49 \\
  19 & 0 & 0 &  & 0.00 & 8.09 \\
   \hline
\end{tabular}
\vspace{3mm}

\begin{tabular}{llllrl}
  \hline
 node\_id & left\_child & right\_child & split\_variable & split\_point & prediction \\
  \hline
  1 & 2 & 3 & smoker & yes & 13.24 \\
  2 & 4 & 5 & bmi & 31.30 & 8.33 \\
  3 & 6 & 7 & region & NE, NW & 31.71 \\
  4 & 8 & 9 & sex & 2.00 & 7.56 \\
  5 & 10 & 11 & region & NE, NW, SE & 9.24 \\
  6 & 12 & 13 & children & 2.50 & 29.45 \\
  7 & 14 & 15 & bmi & 30.10 & 33.69 \\
  8 & 16 & 17 & children & 0.50 & 7.27 \\
  9 & 18 & 19 & children & 1.50 & 7.82 \\
  10 & 0 & 0 &  & 0.00 & 8.69 \\
  11 & 0 & 0 &  & 0.00 & 11.09 \\
  12 & 0 & 0 &  & 0.00 & 28.39 \\
  13 & 0 & 0 &  & 0.00 & 33.71 \\
  14 & 0 & 0 &  & 0.00 & 22.11 \\
  15 & 0 & 0 &  & 0.00 & 41.00 \\
  16 & 0 & 0 &  & 0.00 & 6.79 \\
  17 & 0 & 0 &  & 0.00 & 7.72 \\
  18 & 0 & 0 &  & 0.00 & 7.34 \\
  19 & 0 & 0 &  & 0.00 & 9.04 \\
   \hline
\end{tabular}
}
\end{center}

\vspace{3mm}

\begin{question}{}
Draw the two regression trees from the Insurance Cost Dataset random forest that are represented by these tables. Note: if a variable's value is less than the split point at a particular node, the training sample goes to the left. For categorical variables, if the variable's value is one of the categories listed under ``split point'', the training sample goes to the right.
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Model Parameters}

\subsection{Splitting Criteria}

We already discussed how individual trees are built in Chapter~\ref{chapter:decisiontrees}. One of the choices we made then was which \textbf{splitting criterion} to use in building the tree. The splitting criterion is some way of deciding which variables are ``good'' to split on. For classification trees, the most common criteria are Gini index and information gain (the Gini index is by far the most popular). For regression, \textbf{variance reduction} (the same idea as standard deviation reduction) is the most common criterion.

\vspace{4mm}

\begin{question}{}
The following data come from a Mayo Clinic trial of the drug D-penicillamine for primary biliary cirrhosis (PBC) of the liver. The trial was conducted between 1974 and 1984. The data shown here are for 418 patients who completed the trial. The dataset comes from the \texttt{survival} package in R and contains information on 17 predictors, as well as the follow-up time and outcome (death or censoring) for each patient. Here are Kaplan-Meier curves (see Chapter~\ref{chapter:km}) for four predictors, one of which (bilirubin) I manually binarized. Bilirubin is considered high if it is greater than 1.2 mg/dL.

\begin{center}
\includegraphics[width=0.48\textwidth]{img/rf-surv-example-1.png}
\includegraphics[width=0.48\textwidth]{img/rf-surv-example-2.png}
\includegraphics[width=0.48\textwidth]{img/rf-surv-example-3.png}
\includegraphics[width=0.48\textwidth]{img/rf-surv-example-4.png}
\end{center}

Say you wanted to build a decision tree to predict survival in PBC. You would want to choose splits for which survival looks very \emph{different} on either side of the split. This is analogous to choosing splits that increase the purity of the outcome (for classification) or reduce the variance of the outcome (regression). Speculate on how you might build such a tree. We will discuss the process of constructing \textbf{random survival forests} in much greater detail after we've seen a bit more survival analysis.
\end{question}

\subsection{Creating Split Points}

Each node within a tree signifies a division of one of the predictors into two groups. \textbf{Deterministic splitting} means considering all possible splits and identifying the best one. For a numeric predictor, this involves considering all of the values of the predictor represented in the dataset; there may be as many as $n$ possible values, where $n$ is the number of training samples included in the tree. For a categorical predictor, this involves dividing the possible categories into two \textbf{complementary groups}. For example, the insurance cost dataset in Section~\ref{sect:reginsurance} contains a predictor \texttt{region} with four categories: northeast, southeast, southwest, and northwest. Each split decision must consider all $2^{k-1}-1$ possible divisions\footnote{This comes from taking $2^k$ (total combinations of $k$ categories), subtracting $2$ (all in or all out, neither of which is possible), and then dividing the whole thing by $2$ (because the ordering of the groups doesn't matter). $(2^k - 2)/2 = 2^{k-1}-1$} of those categories, where $k$ is the number of categories. For \texttt{region}, the possible splits are:

\begin{center}
{\small \tt
\begin{tabular}{ll}
Group 1 & Group 2 \\
\midrule
NE, SE, SW & NW \\
NE, SW, NW & SE \\
NE, SE, NW & SW \\
SE, SW, NW & NE \\
NE, SE & NW, SW \\
NE, SW & NW, SE \\
NE, NW & SE, SW \\
\end{tabular}
}
\end{center}

\vspace{3mm}

Deterministic splitting becomes problematic when the number of possible splits is large. In that case, software packages often employ \textbf{random splitting}, in which a predetermined number of possible splits (the exact number is set using a parameter) are randomly chosen from among all the possibilities.

\vspace{5mm}

\begin{question}{}
A known issue with decision trees is their tendency to prefer to split on continuous predictors over discrete predictors. Why do you think this is? It can be avoided, in part, by using random splitting with a fixed number of possible splits.
\end{question}

\subsection{Bag Size and Number of Predictors}

Each tree in a random forest is built using a ``bagged'' sample of $n$ training examples from the original $N$ examples. Typically around $2/3$ of training examples are used per tree, but most software packages include a parameter that allows the user to set this value. In addition, at each split, a tree considers only a randomly-chosen subset of $p$ predictor variables; the default number is usually $\sqrt{P}$ (for classification problems) or $P/3$ (for regression problems), where $P$ is the total number of predictors. In software, this will also be a settable parameter. Usually it's fine to leave these parameters at their default values.

\vspace{5mm}

\begin{question}{}
For an ensemble to be more accurate than any of its individual members, the learners comprising the ensemble must be \emph{accurate} and \emph{diverse}. Accuracy means that the learners must perform better than random on their designated task. Diversity means that the classifiers must make different errors on new data points. How do the parameters $n$ and $p$ impact diversity? How does the way the trees are trained ensure accuracy?
\end{question}

\subsection{Node Size and Tree Depth}

Software packages generally allow the user to control the growth of individual trees within a random forest by specifying node size and tree depth parameters. The \textbf{node size} parameter governs the minimum number of training samples present at a node for a split to be considered. The \textbf{tree depth} parameter governs the maximum number of connections between the root of the tree and one of its leaves. A split at a particular node will only be considered when there is still some impurity in the outcome at that node (see Chapter~\ref{chapter:decisiontrees}) and when:
\begin{enumerate}
\item The current tree depth is less than the maximum allowed tree depth.
\item The number of samples at a node is at least 2x the minimum node size (since a binary split on a smaller node would result in leaves with less than the minimum required node size).
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The Out-of-Bag Error}

The random forest will report a number called the \textbf{out-of-bag (OOB) error} as it runs. To calculate OOB error, each tree makes a prediction for each of the training samples \emph{not} used in its construction. This provides an ongoing estimate of the generalization error of the forest.

Here is an OOB error plot for the random classification forest built on the Wisconsin Breast Cancer dataset:

\begin{center}
\includegraphics[width=0.7\textwidth]{img/oob-classification-wisc.png}
\end{center}
The black line shows the overall OOB error (the percent of OOB points misclassified) after the addition of each new tree. The red line shows the OOB error for points of class 1, which in this case is $B$ (benign), and the green line shows the OOB error for points of class 2, which in this case is $M$ (malignant).

\vspace{4mm}

\begin{question}{}
What does it mean that the green line is so much higher than the red line? What does this tell you about the relative rates of false positives and false negatives for this random forest?
\end{question}

\begin{question}{}
Here is the final \textbf{confusion matrix} for the Wisconsin Breast Cancer random forest.
\vspace{-3mm}
\begin{center}
{\tt
\begin{tabular}{lrrr}
&    B  &  M  & class.error \\
B & 346 & 11 & 0.03081232 \\
M & 16 & 196 & 0.07547170
\end{tabular}
}
\end{center}
\vspace{-3mm}
It is probably more important to avoid false negatives ($M$ tumors that are classified as $B$) than false positives. Speculate on ways in which you could force the forest to produce a lower rate of false negatives, even if it means increasing the number of false positives.
\end{question}

For regression forests, the OOB error is calculated differently. It is defined as the mean square error among the OOB samples. The square root of this error is the average absolute value of the difference between the predicted and actual costs.
\begin{center}
\includegraphics[width=0.7\textwidth]{img/oob-classification-insur.png}
\end{center}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Variable Importance Measures}

One of the main disadvantages of random forests is their lack of clarity around which variables are ``important''. In a regression model, the model output contains hypothesis tests and coefficients for each variable that provide the user with an interpretable importance ranking. Nothing this simple exists for random forests. However, there are some heuristics for ranking variables. These fall into two camps.

\subsection{Impurity-Based Importance}

Trees are built by choosing splits that reduce uncertainty, or impurity, in the outcome. This impurity reduction is a measure of how much splitting on that variable ``helps'' in purifying the outcome. One way to measure the importance of a variable, therefore, is to average the decrease in node impurity across all splits involving that variable, across all trees in the random forest\footnote{Because splits occur at different heights, impurity reduction is typically weighted by how many samples reach a given node.}. This importance measure is called the \textbf{Mean Decrease in Impurity (MDI)}. It works no matter what your outcome is. Its main advantage is that because the reduction in impurity is what is already used to determine the splits, it requires very little additional computation.

\subsection{Permutation-Based Importance}

An alternative way of measuring the importance of variable $j$ is to see how much it affects the predictive accuracy of trees across the whole forest. We assess this using the OOB samples. First the real OOB error is calculated (by running each sample through the trees for which it is OOB). Then the values of variable $j$ across the OOB samples are randomly permuted, and the OOB error is calculated again. We expect the OOB error to go up in the second case by an amount proportional to how important variable $j$ is. We then average this difference for each variable across all trees. This permutation-based importance measure is called the \textbf{Mean Decrease in Accuracy (MDA)}. Again, it works no matter what the outcome is. Its main advantage is that it is more interpretable than MDI.

\vspace{4mm}

\begin{question}{}
Here is a variable importance plot for the Wisconsin Breast Cancer classification forest. Which side is MDI and which is MDA? Which variables are most and least important?
\begin{center}
\includegraphics[width=0.85\textwidth]{img/vimp-classification-wisc.png}
\end{center}
\end{question}

\begin{question}{}
Here is a variable importance plot for the insurance cost dataset regression forest. Which side is MDI and which is MDA? Which variables are most and least important?
\begin{center}
\includegraphics[width=0.85\textwidth]{img/vimp-classification-insur.png}
\end{center}
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{A Note on Software Packages}

As we have seen in Chapter~\ref{chapter:decisiontrees}, there are many different ways to build and optimize decision trees. There are even more ways to build and optimize random forests. This chapter uses the \texttt{randomForest} R package by Andy Liaw, a faithful implementation of the original random forest implementation suggested by Breiman (2003), as well as \texttt{randomForestSRC}, a faster and more recent package by Ishwaran and Kogalur that provides a unified interface for random forest-based classification, regression, and survival analysis. The \texttt{scikit-learn} package in Python provides implementations of random forests for both classification and regression.


\section{Investment and Financing}

\subsection*{Leverage and taxes}

Notation:

$X_t$ - CF from the firm’s assets at time $t$ (independent of leverage), \\
$V_{U,t}$ - value of firm without leverage at $t$, \\
$V_{L,t}$ - value of the firm with leverage at $t$, \\
$D_t$ - value of its debt, \\
$E_t$ - value of its equity, \\
$r_A$ - required rate of return on the firm's assets of the unlevered firm,
$r_L$ - required rate of return on the levered firm,
$r_D$ - required rate of return interest on debt,
$r_E$ - required rate of return on equity, \\
$\tau$ - corporate tax rate.

\subsection*{Leverage without tax shield}

$V = D+E = \sum_{s=1}^{\infty} \frac{(1-\tau)X_s}{(1+r_A)^s} =V_U =\sum_{s=1}^{\infty} \frac{(1-\tau)X_s}{(1+WACC)^s} =V_L$

MM II: Cost of equity with leverage ($D/E$) is:
$r_E=r_A+\frac{D}{E}(r_a-r_D)$ \\
$r_A$ is independent of $D/E$ (leverage),
$r_E$ increases with $D/E$ (assuming riskless debt),
$r_D$ may also increase with $D/E$ as debt becomes risky.

\subsection*{Leverage with tax shield - APV}

$V_L = E+D = V+PVTS-PDVC = APV$, assuming debt is riskless $V_L = E+D = V+PVTS = APV$

\subparagraph*{Leverage with tax shield - WACC}
Assume {\bf Leverage ratio remains constant over time} $w_D = \frac{D_t}{V_{L,t}}$, $\frac{E_t}{V_{L,t}}=w_E$ \\
first, we have:
$r_L = w_D r_D + w_E r_E$ \\
Next, we have:
$(1 + r_L - w_D\tau r_D) V_{L,t}=(1-\tau) X_{t+1} + V_{L,t+1}$ \\

define $WACC = w_D (1-\tau)r_D + w_E r_E$, then: \\

$V_{L,t} = \frac{(1-\tau)X_t + V_{L,t+1}}{1+WACC} = \sum_{s=1}^{\infty} \frac{(1-\tau)X_{t+s}}{(1+WACC)^s}$

\subsection*{WACC with taxes - WACC}

$V_L = \sum_{s=1}^{\infty} \frac{(1-\tau)X_{s}}{(1+WACC)^s} + PVTS$ where $WACC = w_D (1-\tau)r_D + w_E r_E = \frac{D}{D+E} (1-\tau) r_D + \frac{E}{D+E}r_E $

\subsection*{Implementing APV}

1. Find a traded firm with the same business risk: Debt to equity ratio $\frac{D}{E}$, Equity return $r_E$ (by CAPM or APT), Debt return $r_D$, Tax rate $\tau$.
2. Uncover $r_A$ (the discount rate without leverage).
3. Apply $r_A$ to the after-tax cash flow of the project to get $V_U$.
4. Compute PV of debt tax shield.
5. Compute APV.


\section{Quantisation}

\begin{frame}{Lattice Points}

    \begin{itemize}
       \item Certain types of spaces $M$ are special\footnote{Called \emph{``Symplectic toric manifolds''}.}: each has an associated polytope.
       \item Each lattice point inside corresponds to a polynomial.
    \end{itemize}

    \begin{figure}
        \includegraphics[scale=0.25]{resources/polynomials.png}
    \end{figure}

\end{frame}

\begin{frame}{Quantisation Dimension}

    \begin{itemize}
    \item Degree $k$ of the polynomial analogous to quantised ``energy'' of the system.
    \item Quantisation dimension $M$ equals the lattice point count (how many wavefunctions there are).
    \end{itemize}

    \begin{figure}
        \includegraphics[scale=0.25]{resources/polynomials.png}
    \end{figure}

\end{frame}

\begin{frame}{Combinatorics}

    \begin{itemize}
       \item Useful to view the polytope as an intersection of half-spaces.
    \end{itemize}

    \begin{figure}
        \includegraphics[scale=0.25]{resources/half-spaces.png}
    \end{figure}

\end{frame}


\vsssub
\subsubsection{~$S_{is}$: Diffusive scattering by sea ice (simple)} \label{sec:IS1}
\vsssub

\opthead{IS1}{\ws/NRL}{S. Zieger}

\noindent
The non-conservative effect of ice on waves has been implemented in switches
{\code IC1} through {\code IC3} (see \para\ref{sec:ICE1}--\ref{sec:ICE3}). The
conservative effect of sea ice has been implemented in switch {\code IS1} and
represents a simple form of scattering. It is assumed that the floe size is
smaller than the grid size and that a fraction $\alpha_{ice}$ of the incoming
wave energy is scattered isotropically. The fraction is determined from sea ice
concentration ICE using a simple linear transfer function
\begin{equation}\label{eq:IS101}
     \alpha_{ice} = \max\left \{0 , C_1\ \mathrm{ICE} + C_2 \right \} \:\:\: .
\end{equation}

\noindent
The coefficients $C_1$ and $C_2$ are customizable through namelist {\F SIS1}
with namelist parameters {\code ISC1} and {\code ISC2}. At each discrete
frequency and direction the wave energy is reduced by the amount of
$\alpha_{ice}$ and redistributed to all direction in the same discrete
frequency to conserve energy.


\section{Theoretical Foundations}
\label{sec:Theoretical_Foundations}
This section summarizes the theory and formulas necessary to understand the following experiments in section \ref{sec:Evaluation}.

\subsection{Measurement Errors}
\label{subsec:Measurement_Errors}
Measurement errors are inevitable. They may result from the influence of the test conditions, the measuring instruments themselves or human errors. The deviation of a measured value from its true value affects the result. Thus, the result also deviates from its true value. The error analysis tries to estimate the effect of a measurement error on the result.

\subsubsection{Types of Measurement Errors}
\label{subsubsec:Types_of_Measurement_Errors}
Generally, one differentiates two types of measurement errors: random and systematic errors. Random measurement errors cause the measured values to fluctuate around the true value. They can be decreased by repeating the same measurement several times. Systematic errors are a result of the test conditions and the measuring instruments. Most of the spotted systematic errors can be corrected. Usually, the problem is that they are not easy to spot.

\subsubsection{Precision of Measurement Errors}
\label{subsubsec:Precision_of_Measurement_Errors}
Every physical measurement needs an according error analysis to be meaningful. Since the measurement error is just an estimate, it is sufficient to round the error to one significant digit.

\subsection{Formulas}
\label{subsec:Formulas}
The following formulas are important to estimate measurement errors.

\subsubsection{Arithmetic Mean}
\label{subsubsec:Arithmetic_Mean}
To determine the arithmetic mean value from several measurements ($x_1, x_2, x_3, ...\ x_n$), the following equation is used:
\begin{equation}
\overline x=\frac{1}{n}\left(\ \sum_{i=1}^{n} x_i\ \right)
\label{eq:Arithmetic_Mean}
\end{equation}

The error of the arithmetic mean value is calculated as follows:
\begin{equation}
s_{\overline x}=\sqrt{\frac{\sum\limits_{i=1}^{n}\left(x_i - \overline x\right)^2}{n\cdot(n-1)}}
\label{eq:Arithmetic_Mean_Error}
\end{equation}

Result:
\begin{equation}
x=\overline x \pm s_{\overline x}
\label{eq:Arithmetic_Mean_Result}
\end{equation}

This is only valid, if all the measurements $x_i$ have the same accuracy and are independent of each other.

\subsubsection{Weighted Arithmetic Mean}
\label{subsubsec:Weighted_Arithmetic_Mean}
When a physical quantity has been determined with different measuring methods, the results may have different accuracies ($x_i={\overline{x_i}}\pm s_{\overline{xi}}$). To determine the weighted arithmetic mean value the following equation is used:
\begin{equation}
\overline x=\frac{\sum\limits_{i=1}^{n}g_{\overline {xi}}\cdot x_i}{\sum\limits_{i=1}^{n}g_{\overline {xi}}} \qquad \text{with the respective weights} \qquad g_{\overline {xi}}=\frac{1}{{s_{\overline {xi}}}^2}
\label{eq:Weighted_Arithmetic_Mean}
\end{equation}

The error of the weighted arithmetic mean value is calculated as follows:
\begin{equation}
s_{\overline x}=\frac{1}{\sqrt{\sum\limits_{i=1}^{n}g_{\overline {xi}}}}
\label{eq:Weighted_Arithmetic_Mean_Error}
\end{equation}

Result:
\begin{equation}
x=\overline x \pm s_{\overline x}
\label{eq:Weighted_Arithmetic_Mean_Result}
\end{equation}

Weighting requires the errors to be random.

\subsubsection{Error Theory}
\label{subsubsec:Error_Theory}
The standard deviation $\sigma$ provides information on the spread of the measured values $x_i$. To calculate the experimental standard deviation from $n$ measurements (finite) the following equation is used:
\begin{equation}
s=\sqrt{\frac{\sum\limits_{i=1}^{n}\left(x_i - \overline x\right)^2}{n-1}}
\label{eq:Experimental_Standard_Deviation}
\end{equation}

The calculated experimental standard deviation $s$ converges to $\sigma$ for $n\to\infty$.

\subsubsection{Regression}
\label{subsubsec:Regression}
Some experiments seek to prove a physical law or to determine parameters of a functional relationship between physical quantities. To accomplish this for a given set of measured value pairs ($x_i$, $y_i$), the parameters ($a_0$, $a_1$, ...) of a given law $f(x, a_0, a_1, ...)$ are adjusted, so that the best possible agreement between theory and experiment is achieved.

According to the error theory of Gauss, the most probable set of parameters is found, when the sum of the squared deviations becomes minimal:
\begin{equation}
\chi^2(a_1, a_2, ...\ a_m)=\sum_{i=1}^{n}\left\{\frac{[y_i-f(x_i, a_1, a_2, ...\ a_m)]^2}{{\sigma_i}^2}\right\}=\text{Minimum}
\end{equation}

The following equation is used estimate the error from the measurements and the fit:
\begin{equation}
\sigma=\sqrt{\frac{\sum\limits_{i=1}^{n}[y_i-f(x_i, a_1, a_2, ...\ a_m)]^2}{n-m}}
\end{equation}

\subsubsection{Error Propagation}
\label{subsubsec:Error_Propagation}
In the majority of cases, physical experiments do not directly provide the wanted measurement result. Usually, one ends up with a function $R$ which depends on several variables:
\[
R=R(x,\ y,\ z,\ ...)
\]
The arguments $x,\ y,\ z,\ ...$ are either directly measured quantities or values from textbooks in the form:
\[
x={\overline{x}}\pm s_{\overline{x}},\qquad y={\overline{y}}\pm s_{\overline{y}},\qquad z={\overline{z}}\pm s_{\overline{z}},\qquad ...
\]
The mean value of $R$ ($\overline R$) can be determined by inserting the mean values of the arguments:
\begin{equation}
\overline{R}=R(\overline{x},\ \overline{y},\ \overline{z},\ ...)
\label{eq:Error_Propagation_Mean}
\end{equation}
The mean absolute error $s_{\overline R}$ is calculated using the following equation:
\begin{equation}
s_{\overline{R}}=\sqrt{\left(\frac{\partial R}{\partial x}\Biggr|_{\overline R}\cdot s_{\overline{x}}\right)^2 + \left(\frac{\partial R}{\partial y}\Biggr|_{\overline R}\cdot s_{\overline{y}}\right)^2 + \left(\frac{\partial R}{\partial z}\Biggr|_{\overline R}\cdot s_{\overline{z}}\right)^2 +\ \ ...}
\label{eq:Error_Propagation}
\end{equation}
Result:
\begin{equation}
R=\overline R \pm s_{\overline R}
\label{eq:Error_Propagation_Result}
\end{equation}


\documentclass{article}

% To change the font size of text in \section
\makeatletter
\renewcommand\section{\@startsection{section}{1}{\z@}%
                                  {-3.5ex \@plus -1ex \@minus -.2ex}%
                                  {2.3ex \@plus.2ex}%
                                  {\normalfont\large\bfseries}}
\makeatother

\usepackage[pdftex]{graphicx}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{mathtools}

\newcommand{\bigslant}[2]{{\raisebox{.1em}{$#1$\!}\left/\raisebox{-.2em}{$#2$}\right.}}

% Horizantal lines lines in title page
\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}

\newtheorem{thm}{Theorem}[section]

\theoremstyle{definition}
\newtheorem{defn}{Definition}[section]
\newtheorem{exr}{Exercise}[section]

\theoremstyle{remark}
\newtheorem{rmk}{Remark}[section]

\renewcommand{\familydefault}{\sfdefault}

\usepackage[left=2.5cm,right=2.5cm,top=2cm,bottom=2cm]{geometry}

\begin{document}
\input{Title}
\input{MetricSpaces}
\input{DetectiveProblem}
\input{LengthInMetricSpaces}
\input{LengthOfSmoothCurves}
\input{LengthEnergy}
\input{CauchySchwarzInequality}
\input{EnergyMinimization}
\input{CPoE_EuclideanSpace}
\input{CPoE_Sphere}
\input{Metric_Graphs}
\end{document}


\subsection{Squares}

\subsubsection{Area of a square}

\(A=r^2\)

\subsubsection{Circumference of a square}

\(C=2r\)

\subsubsection{Angles in a square}

Angles in a square sum to \(2\pi \).


\documentclass[10pt]{report}

\usepackage{enumerate} % for enumerate counter
\usepackage{subcaption} % for subfigures
\usepackage{amsthm} % for QED
\usepackage{mathtools} % for delimiter

\usepackage{listings} % for code
\lstset{
	language=R,
	basicstyle=\footnotesize\ttfamily,
	numbers=none,
	stepnumber=1,
	numbersep=8pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	frame=single,
	tabsize=2,
	captionpos=t,
	breaklines=true,
	breakatwhitespace=false
}

\usepackage{float} % for figure [H]
\usepackage{booktabs} % for tabular
\usepackage{caption} % for \caption*
\usepackage[export]{adjustbox} % for valign=t
\usepackage{array} % for column type m
\usepackage{verbatim}
\usepackage{graphicx}
%\graphicspath{ {imgs/} }
\usepackage{fancyhdr}
\usepackage{amssymb}
\usepackage{amsmath}

%%%%%%Pagination
\setlength{\topmargin}{-.3 in}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textheight}{9.in}
\setlength{\textwidth}{6.5in}

%Cover
\newcommand{\hwTitle}{Homework \#6}
\newcommand{\hwCourse}{Applied Statistics}
\newcommand{\hmClassInstructor}{Professor Lulu Kang}

\title{
	\vspace{2in}
	\textmd{\textbf{\hwCourse\\\hwTitle}}\\
	\vspace{0.3in}\large{\textit{\hmClassInstructor}}
	\vspace{3in}
}
\author{\textbf{Zhihao Ai}}
\date{}

%Header
\pagestyle{fancy}
\fancyhead[L]{Zhihao Ai}
\fancyhead[C]{Math 484}
\fancyhead[R]{Homework 6}
%%%%%%

%Global settings
%\everymath{\displaystyle}
\setlength\parindent{0pt}

%Custom commands
\newcommand{\ds}{\displaystyle}
\newcommand{\ts}{\textstyle}

\newcolumntype{N}{>$ c <$}
\newcolumntype{M}[1]{>{\centering\arraybackslash $}m{#1}<{$}}

\newcommand{\abs}[1] {\left| #1 \right|}

\DeclarePairedDelimiter\autoparen{(}{)}
\newcommand{\pa}[1]{\autoparen*{#1}}

\newcommand{\var} {\text{var}}

\newcommand{\m}[1] {\mathbf{#1}}

\begin{document}

\maketitle

\section*{Problem 1}
(Ex. 14.9) \textbf{Performance ability} + (Ex. 14.10(a))
\begin{enumerate}[a.]
	\item
	Find the maximum likelihood estimate of $\beta_0$ and $\beta_1$. State the fitted response function.
	\lstinputlisting{p1/9a.txt}
	The fitted response function is
	\[
	\hat{\pi} = \frac{\exp(-10.309 + 0.019X)}{1 + \exp(-10.309 + 0.019X)}
	\]
	The maximum likelihood estimate of $\beta_0$ and $\beta_1$ are $b_0 = -10.309$ and $b_1 = 0.019$.

	\item
	Obtain a scatter plot of the data with both the fitted logistic response function from part (a) and a lowess smooth superimposed. Does the fitted logistic response function appear to fit well?
	\begin{figure}[H]
		\centering
		\includegraphics[width=.5\linewidth]{p1/9b.png}
	\end{figure}
	The fitted logistic response function appears to fit well.

	\item
	Obtain $\exp(b_1)$ and interpret this number.

	$\exp(b_1) = 1.0191$. The odds ratio $\widehat{OR} = \exp(b_1) = 1.0191$ means that the odds of being able to perform in a task group are estimated to increase by about 1.9 percent with each unit increase in the score of employee's emotional stability.

	\item
	What is the estimated probability that employees with an emotional stability test score of 550 will be able to perform in a task group?

	The estimated probability is $\hat{\pi}(550) = \frac{\exp(-10.309 + 0.019\cdot 500)}{1 + \exp(-10.309 + 0.019\cdot 500)} = 0.308$.

	\item
	Estimate the emotional stability test score for which 70 percent of the employees with this test score are expected to be able to perform in a task group.

	Solving
	\[
	\log\pa{\frac{0.7}{1-0.7}} = -10.309 + 0.019X
	\]
	we have $X=587$. So with the emotional stability test score being 587, 70 percent of the employees are expected to be able to perform in a task group.

	\item
	Fit a probit mean response function (14.12) to the data. Quantitatively compare the fit here with the logistic fit obtain in part (a). What do you conclude?
	\lstinputlisting{p1/9f.txt}
	The probit mean response function is
	\[
	\hat{\pi} = \Phi(-6.374 + 0.012X)
	\]
	The coefficients of $X$ are close to one another in the two functions, but the intercept is larger in the probit mean response function.
\end{enumerate}

\section*{Problem 2}
(Ex. 14.13) \textbf{Car purchase}
\begin{enumerate}[a.]
	\item
	Find the maximum likelihood estimates of $\beta_0$, $\beta_1$ and $\beta_2$. State the fitted response function.
	\lstinputlisting{p2/13a.txt}
	The fitted response function is
	\[
	\hat{\pi} = \frac{\exp(-4.739 + 0.068X_1 + 0.599X_2)}{1 + \exp(-4.739 + 0.068X_1 + 0.599X_2)}
	\]
	The maximum likelihood estimate of $\beta_0$, $\beta_1$ and $\beta_2$ are $b_0 = -4.739$, $b_1 = 0.068$ and $b_2 = 0.599$.

	\item
	Obtain $\exp(b_1)$ and $\exp(b_2)$ and interpret these numbers.

	$\exp(b_1) = 1.070079$ and $\exp(b_2) = 1.819627$. The odds ratio $\widehat{OR}_1 = \exp(b_1) = 1.070079$ means that, with the current age of the oldest family automobile($X_2$) fixed, the odds of the family purchasing a new car are estimated to increase by about 7 percent with each unit increase in annual family income($X_1$). The odds ratio $\widehat{OR}_2 = \exp(b_2) = 1.819627$ means that, with the annual family income($X_1$) fixed, the odds of the family purchasing a new car are estimated to increase by about 82 percent with each unit increase in the current age of the oldest family automobile($X_2$).

	\item
	What is the estimated probability that a family with annual income of \$50 thousand and an oldest car of 3 years will purchase a new car next year?

	The estimated probability is $\hat{\pi}(50, 3) = \frac{\exp(-4.739 + 0.068\cdot 50 + 0.599\cdot 3)}{1 + \exp(-4.739 + 0.068\cdot 50 + 0.599\cdot 3)} = 0.61254$.
\end{enumerate}

\section*{Problem 3}
(Ex. 14.12) \textbf{Toxicity experiment}
\begin{enumerate}[a.]
	\item
	Plot the estimated proportions $p_j = Y_{.j} / n_j$ against $X_j$. Does the plot support the analyst's belief that the logistic response function is appropriate?
	\begin{figure}[H]
		\centering
		\includegraphics[width=.5\linewidth]{p3/12a.png}
	\end{figure}
	It supports the analyst's belief that the logistic response function is appropriate.

	\item
	Find the maximum likelihood estimates of $\beta_0$ and $\beta_1$. State the fitted response function.
	\lstinputlisting{p3/12b.txt}
	The fitted response function is
	\[
	\hat{\pi} = \frac{\exp(-2.644 + 0.674X)}{1 + \exp(-2.644 + 0.674X)}
	\]
	The maximum likelihood estimate of $\beta_0$ and $\beta_1$ are $b_0 = -2.644$ and $b_1 = 0.674$.

	\item
	Obtain a scatter plot of the data with the estimated proportions from part (a), and superimpose the fitted logistic response function from part (b). Does the fitted logistic response function appear to fit well?
	\begin{figure}[H]
		\centering
		\includegraphics[width=.5\linewidth]{p3/12c.png}
	\end{figure}
	The fitted logistic response function appears to fit well.

	\item
	Obtain $\exp(b_1)$ and interpret this number.

	$\exp(b_1) = 1.962056$. The odds ratio $\widehat{OR} = \exp(b_1) = 1.962056$ means that the odds of insects dying due to the toxic substance are estimated to increase by about 96 percent with each unit increase in the dose level.

	\item
	What is the estimated probability that an insect dies when the dose level is $X=3.5$?

	The estimated probability is $\hat{\pi}(3.5) = \frac{\exp(-2.644 + 0.674X)}{1 + \exp(-2.644 + 0.674X)} = 0.429$.

	\item
	What is the estimated median lethal dose---that is, the dose for which 50 percent of the experimental insects are expected to die?

	Solving
	\[
	\log\pa{\frac{0.5}{1-0.5}} = -2.644 + 0.674X
	\]
	we have $X=3.92$. So with the does level being 3.92, 50 percent of the experimental insects are expected to die.
\end{enumerate}

\end{document}


\documentclass[12pt]{article}
\addtolength{\textwidth}{20mm}
\addtolength{\hoffset}{-10mm}

\def\bea{\begin{eqnarray}}
\def\eea{\end{eqnarray}}
\def\Eq#1{Eq(\ref{#1})}
\def\Re{\mathrm{Re}}

\title{Error estimate of $z$-coordinate in 3D Single Molecule Localization}
\author{Tung-Han Hsieh}

\begin{document}
\maketitle

The relation between $z$-coordinate of the molecules and the width in
$x$, $y$ direction is through the calibration curve (the defocusing curve):
\bea\label{calb}
w_{x,y}(z) = w_0\Re\left(\sqrt{ 1 +
	\left(\frac{z-c_{x,y}}{d_{x,y}}\right)^2 +
	A_{x,y}\left(\frac{z-c_{x,y}}{d_{x,y}}\right)^3 +
	B_{x,y}\left(\frac{z-c_{x,y}}{d_{x,y}}\right)^4 }\right)
\eea
where $w_0$, $A_{x,y}$, $B_{x,y}$, $c_{x,y}$, and $d_{x,y}$ are calibration
parameters determined from experiments. In general these parameters are complex
numbers (except the parameter $w_0$). The $z$-coordinate of the molecules
can be obtained via solving $z$ in \Eq{calb} with respect to the
fitted width $w_x$ and $w_y$ from experimental measurements.

There are two ways to determine the $z$ coordinate. The first one is to
solve $z_x$ and $z_y$ for the input of $w_x$ and $w_y$, and then take the
average of $z_x$ and $z_y$. The second one is to solve $w_x/w_y$ for $z$.
We describe the error estimate of $z$ through the input of $w_x$, $w_y$
and their errorbars $\delta w_x$, $\delta w_y$ for both methods.

\section{Method 1: average of $z_x$ and $z_y$}

For convenience, we first consider that all calibration parameters are
real numbers. We define the variable $u_{x,y}$ as:
\bea\label{u(z)}
u(z) = \left(\frac{z-c}{d}\right)^2 +
	A\left(\frac{z-c}{d}\right)^3 +
	B\left(\frac{z-c}{d}\right)^4
\eea
Here we have suppressed the labels $x$ and $y$ for simplicity, which will
be put back when necessary. Then \Eq{calb} can be rewritten as
\bea
w(z) = w_0 \sqrt{1+u(z)}
\eea
Suppose that there is no errorbars in the calibration parameters.
The only errorbar is $\delta w$, which comes from the fitting
for $w(z)$. Then the relation between $\delta w$ and $\delta z$ is
\bea\label{dw}
\delta w = \frac{w_0^2}{2 |w(z)|} \delta u
\eea
where
\bea\label{du}
\delta u = \sqrt{ \left[2\Bigl(\frac{z-c}{d}\Bigr)\right]^2 +
		  \left[3A\Bigl(\frac{z-c}{d}\Bigr)^2\right]^2 +
		  \left[4B\Bigl(\frac{z-c}{d}\Bigr)^3\right]^2 }\delta z
\eea
Putting \Eq{dw} and \Eq{du} together one can solve $\delta z$ from $\delta w$:
\bea\label{dz-dw}
\delta z = \frac{2 |w(z)|}{w_0^2} \delta w \left\{
	\left[2\Bigl(\frac{z-c}{d}\Bigr)\right]^2 +
	\left[3A\Bigl(\frac{z-c}{d}\Bigr)^2\right]^2 +
	\left[4B\Bigl(\frac{z-c}{d}\Bigr)^3\right]^2 \right\}^{-1/2}
\eea
Therefore, for the input of calibration parameters, $w_{x,y}(z)$,
and $\delta w_{x,y}$, we can obtain $\delta z_{x,y}$. Finally, errorbar
$\delta z_a$ of the average $z_x$ and $z_y$ is
\bea\label{z_a}
\delta z_a = \frac{1}{2} \sqrt{(\delta z_x)^2 + (\delta z_y)^2}
\eea
Now for in general the parameters $A$, $B$, $c$, and $d$ may be complex
numbers, we modify \Eq{dz-dw} as
\bea\label{dz-dw-abs}
\delta z = \frac{2 |w(z)|}{w_0^2} \delta w \left\{
	\left|2\Bigl(\frac{z-c}{d}\Bigr)\right|^2 +
	\left|3A\Bigl(\frac{z-c}{d}\Bigr)^2\right|^2 +
	\left|4B\Bigl(\frac{z-c}{d}\Bigr)^3\right|^2 \right\}^{-1/2}
\eea
That is, take the absolute square of the three terms inside the overall
square root, so that the resulting $\delta z_{x,y}$ is real and positive
quantity. Then \Eq{z_a} can be used to obtain the final errorbar of $z$
coordinate.

\section{Method 2: solving for $w_x(z)/w_y(z)$}

For $v(z) = w_x(z)/w_y(z)$, its errorbar $\delta v$ is
\bea\label{dv1}
\delta v = v\sqrt{ \left(\frac{dw_x}{w_x}\right)^2 +
		   \left(\frac{dw_y}{w_y}\right)^2 }
\eea
Here we understood that $v$, $w_x$, and $w_y$ are all functions of $z$.
On the other hand, $v(z)$ can also be written in the defocusing relation
(suppose that all calibration parameters are real for this moment):
\bea
v(z) = \frac{\sqrt{1+u_x(z)}}{\sqrt{1+u_y(z)}}
\eea
where $u_x(z)$ and $u_y(z)$ are defined in \Eq{u(z)}. Thus we can
obtain the errorbar of $v$ as
\bea\label{dv2}
\delta v = \frac{1}{2} v \sqrt{
	\left(\frac{\delta u_x}{1+u_x}\right)^2 +
	\left(\frac{\delta u_y}{1+u_y}\right)^2 }
\eea
where $\delta u_x$ and $\delta u_y$ are defined as \Eq{du}. Therefore
\bea
\left(\frac{\delta v}{v}\right)^2 &=& \frac{(\delta z)^2}{4}\Biggl\{
	\left|\frac{1}{1+u_x}\right|^2\left[
	4\left|\frac{z-c_x}{d_x}\right|^2 +
	9|A_x|^2\left|\frac{z-c_x}{d_x}\right|^4 +
	16 |B_x|^2\left|\frac{z-c_x}{d_x}\right|^6 \right]
\nonumber\\ && \hspace{8mm} +
	\left|\frac{1}{1+u_y}\right|^2\left[
	4\left|\frac{z-c_y}{d_y}\right|^2 +
	9|A_y|^2\left|\frac{z-c_y}{d_y}\right|^4 +
	16 |B_y|^2\left|\frac{z-c_y}{d_y}\right|^6 \right]
	\Biggr\}
\nonumber\\ &=&
	\left(\frac{dw_x}{w_x}\right)^2 + \left(\frac{dw_y}{w_y}\right)^2
\eea
So the estimated errorbar of $z$ coordinate $\delta z$ can be calculated.

\end{document}


\documentclass[11pt]{amsart}
\usepackage{geometry}
\geometry{letterpaper}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{epstopdf}
\usepackage{fancyhdr}
\usepackage{tikz}
\usetikzlibrary{dsp,fit}

\newcommand{\nom}{\mathrm{nom}}
\newcommand{\cmd}{\mathrm{cmd}}
\newcommand{\filt}{\mathrm{filt}}
\newcommand{\meas}{\mathrm{meas}}
\newcommand{\aero}{\mathrm{aero}}
\newcommand{\motor}{\mathrm{motor}}
\newcommand{\prop}{\mathrm{prop}}

\begin{document}

\subsection{Motor model}

The motor model in the simulator is described by the following
equations:
%
\begin{eqnarray}
I_{\prop} \dot \omega &=& \tau_{\motor} - \tau_{\aero} \\
\dot{\omega}_{\cmd,f} &=& 2 \pi f_{\cmd} (\omega_{\cmd} - \omega_{\cmd,f}) \\
\dot{\omega}_{\meas,f} &=& 2 \pi f_{\meas} (\omega - \omega_{\meas,f}) \\
\tau_{\motor} &=& k_{\omega}_p \left( \omega_{\cmd,f} - \omega_{\meas,f} \right) +
k_{\omega}_i \int_0^t dt' \left( \omega_{\mathrm{cmd,f}} - \omega_{\meas,f} \right) \\
\tau_{\mathrm{aero}} &=& k_P \omega^2
\end{eqnarray}
where $I_{\mathrm{prop}}$ is the moment-of-inertia of the propeller
and rotor, $\omega$ is the angular rate of the motor, $f_{\cmd}$ is
the cut-off frequency of the speed command filter, $f_{\meas}$ is the
cut-off frequency of the speed measurement filter, $k_{\omega}$ is the
gain of the speed control loop, and $k_P$ is the torque constant of
the propellers.  Currently, in the simulator we have
$I_{\mathrm{prop}} = 1.1$ kg m$^2$, $k_P = 0.024$ N-m-s$^2$/rad$^2$,
$k_{\omega}_p = 36.2$ N-m-s/rad, $k_{\omega}_i = 36.2$ N-m-s$^2$/rad.
The typical operating point of the rotors is around,
$\omega_{\mathrm{nom}} = 130$ rad/s.

Based on the motor model equations, the transfer function for the
motor's angular rate is:
\begin{eqnarray}
\frac{\Omega(s)}{\Omega_{\mathrm{cmd}}(s)}
&=& \frac{H(s)}{I_{\mathrm{prop}} s + H(s) + 2 k_P \omega_{\mathrm{nom}}} \\
H(s) &=& \frac{\omega_m}{s + \omega_m} (k_{\omega}_p + k_{\omega}_i / (s - p))
\end{eqnarray}
The DC gain of the motors is
\begin{equation}
\frac{\Omega(0)}{\Omega_{\mathrm{cmd}}(0)} =
\frac{k_{\omega}_p + k_{\omega}_i / p}
{k_{\omega}_p + k_{\omega}_i / p + 2 k_P \omega_{\mathrm{nom}}}
\end{equation}
Here $p = 2 \pi 0.03$ rad/s is a ``fake'' pole that we put into the
simulator so the motor transfer function doesn't fight the stacking
controller.

And thus, we can reasonably describe the overall motor moment, and
thrust, transfer functions by a second order sytem:
\begin{equation}
\frac{M_{\mathrm{motor}}(s)}{M_{\mathrm{cmd}}(s)} =
\frac{\omega_0^2}{s^2 + 2 \zeta \omega_0 s + \omega_0^2}
\end{equation}
For the motors currently in the simulator $\omega_0 = 2 \pi 10$ rad/s and
$\zeta = 1$.

\end{document}


\chapter{Relationship between Algorithms and Complexity}

\section{Problems and Algorithms}
The concept of a "problem" colligates a set of questions making reference to the dynamic or structural particularities of an entity- processes or objects- that acquires pinpointed, defined answers, whose accuracy can be rigorous demonstrated. The units create the universe of the problem. That segment of the problem, which is formerly familiar (known) represents the facts of the problem and the answers to the question shape the solutions. ~\cite{giumale2004introducere};
\\
Making use of a conventional inscription in order to specify an effective approach, taking into account the computational individuality of a calculus machine, it's called an \textbf{algorithm}.

An algorithm that can be accepted by a calculus machine consists of a limited amount of instructions with accurate signification, that can be implemented by the machine in a restricted period of time in order to determine the answer of a problem.

The definition of the algorithm reveals the crucial distinction between a function and an algorithm, which is, to be more specifically, the distinction between a functional representation of a problem and its effective solution. ~\cite{giumale2004introducere}

In this chapter, we will define various metrics for estimating an algorithm's performances in regard to a specific metric, such as: \textbf{RM1}, \textbf{RM2} with enhanced variations \textbf{ERM1}, \textbf{ERM2}.

\section{From algorithm to Normalized rComplexity}
Algorithms' Complexity Theory studies the static and the dynamic performances of mechanical-solvable problems. Static performances aim the clarity of the evaluated algorithm or the abstraction level imposed by used operations, while, more interesting and comparable, the dynamic performances imposes a calculus regarding the amount of resources (e.g. total execution time, peak memory usage) required for executing the algorithm. ~\cite{giumale2004introducere}; Theoretical, such metrics can be exactly calculated for any given algorithm.

\begin{remark}
    Mechanical-solvable problems inevitably makes us think to the related field in physics. An intuitive approach on the mechanical-solvable process is accessible
    to understanding when considering massive motion elements rather than transistors in technologies as advanced as 7nm.

    An example of an early mechanical solution (exhibited in 1862 at The International Exhibition in South Kensington, UK) to a mechanical-solvable problems is the Babbage's design for the difference engine, which is an automatic mechanical calculator designed to tabulate polynomial functions.

    This was a solution for approximation some mathematical functions frequently used, including logarithmic and trigonometric functions, which could be approximated by polynomials.

    \begin{figure}[H]
        \centering
        \includegraphics[width=0.5\textwidth]{BabbageDifferenceEngine}
        \caption{A computing machine built from Babbage's design of a difference engine exhibited at The London Science Museum.}
    \end{figure}

\end{remark}

While the dynamic performances of an algorithm are precisely calculable, due to the high complexity \textit{(by high complexity it is denoted that it is extremely difficult to exactly compute it)} of most computer programs, it is often more suitable an approximation. Such resemblance is provided by the classical model, with drastic compromise compared to the real behavior in a non-asymptotic analysis. A in-between solution for approximating the dynamic performances of an algorithm is provided by \textbf{Normalized rComplexity Calculus Model}, whose aim is to close the gap between the architecture of the computing machine and a \textit{generic-written} algorithm for solving a problem.

\section{Estimating computational time based on Normalized rComplexity}
Let an arbitrary algorithm $Alg$ characterized by the complexity function $f$ with a variable input dimension $n \in \mathcal{N}^{*}$. Consider that the input size is bounded such that $n \in [n_{min}, n_{max}]$. \\
We aim to define various metrics for approximation an average computational time required based on the size of the input and the algorithm's complexity function  $T(n_{min}, n_{max})$.\\

\begin{definition}
    \textbf{RM1}

    Defined as a metric for time estimation (capable of generalization to any other estimators) based on arithmetic mean in Normalized rComplexity model is defined as follows:
    \[  T(n_{min}, n_{max}) = \dfrac{\sum\limits_{n=n_{min}}^{n_{max}} g_{1}(n)}{n_{max} - n_{min} + 1}  \]
\end{definition}

\begin{definition}
    \textbf{RM2}

    Defined as a metric for time estimation (capable of generalization to any other estimators) based on Mean-Value Theorem(Lagrange) using integrals in Normalized rComplexity model is defined as follows:
    \[  T(n_{min}, n_{max}) = \dfrac{\int\limits_{n_{min}}^{n_{max}} g_{1}(n) dn}{n_{max} - n_{min}}  \]
\end{definition}

The previous two metrics are tailored for systems where the input size is bounded but there is no additional knowledge regarding the weights and probabilities of occurrence. If this information is available, we can redefine the previous metrics using the acquired data.

\begin{definition}
    \textbf{ERM1}, an enhanced metric for time estimation based on arithmetic mean in Normalized rComplexity model is defined as follows:
    \[  T(n_{min}, n_{max}) = \sum\limits_{n=0}^{f} p_{n} \cdot g_{1}(n + n_{min})  \]
    where:
    \begin{itemize}
        \item $p_{0}$ is the weight associated with $n_{0} = n_{min}$
        \item $p_{1}$ is the weight associated with $n_{1} = n_{min + 1}$
        \item $p_{f}$ is the weight associated with $n_{f} = n_{max}$
    \end{itemize}
    and $f = max - min + 1$.
\end{definition}

\begin{definition}
    \textbf{ERM2}, an enhanced metric for time estimation based on Mean-Value Theorem(Lagrange) using integrals in Normalized rComplexity model is defined as follows:
    \[  T(n_{min}, n_{max}) =\sum\limits_{k=0}^{f-1} p_{k} \cdot \int\limits_{n_{k}}^{n_{k+1}} g_{1}(n) dn  \]
    where:
    \begin{itemize}
        \item $p_{0}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{0}, n_{1}]$
        \item $p_{1}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{1}, n_{2}]$
        \item $p_{f-1}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{f-1}, n_{f}]$
    \end{itemize}
    and $f = max - min + 1$, and $n_{0} = n_{min}, n_{f} = n_{max}$.
\end{definition}

\section{Comparing algorithms asymptotic performances}

This section aims to compare two generic Algorithms ($Alg1$, $Alg2$) time-based performances based on associated Big r-Theta class from Normalized rComplexity model, by asymptotically correlating the characterized complexity functions $f, f'$.

For specific application of this theoretical work, please refer to the following chapter.

Let $f \in \Theta_{1}(g_{1}(n))$ and $f' \in \Theta_{1}(g'_{1}(n))$.
We can therefore compare the time-based performances of the two algorithms by evaluating:  \[\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)}\]

\begin{remark}
    \[\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = \lim_{n\to\infty} \dfrac{f(n)}{f'(n)}\]
\end{remark}

\begin{definition}
    Let $f \in \Theta_{1}(g_{1}(n))$ and $f' \in \Theta_{1}(g'_{1}(n))$. $f$ is part of a smaller class than $f'$ iff $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = 0$.
\end{definition}
\begin{lemma}
    If  $ \lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = 0 $ , then $Alg1$ will terminate faster than $Alg2$ for any input size $n \geq n_{0}$, with the possibility of computing $n_{0} \in \mathcal{N}^{*}$.
\end{lemma}
\begin{proof}
    $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = 0 \Rightarrow g_{1}(n) < g'_{1}(n)\ \forall n \geq n_{0} \Rightarrow f(n) < f'(n) \ \forall n \geq n_{0}'$
\end{proof}

\begin{definition}
    Let $f \in \Theta_{1}(g_{1}(n))$ and $f' \in \Theta_{1}(g'_{1}(n))$. $f$ is part of the same class as $f'$ iff $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = r$ and $r \in \mathcal{R}_{+}$.
    If $f$ is part of the same class as $f'$, we can distinguish the following cases:
    $$
    \lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} =
    \begin{cases}
        x \in (0,1), f\ has\ a\ smaller\ constant\ than\ f' \\
        1, f\ has\ the\ same\ constant\ as\ f'\\
        y \in (1,\infty), f\ has\ a\ bigger\ constant\ than\ f'
    \end{cases}
    $$
\end{definition}

\begin{lemma}
    If  $ \lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = r \in (0,1) $ , then $Alg1$ will terminate faster than $Alg2$ for any input size $n \geq n_{0}$, with the possibility of computing $n_{0} \in \mathcal{N}^{*}$.
\end{lemma}
\begin{proof}
    $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} < 1 \Rightarrow g_{1}(n) < g'_{1}(n)\ \forall n \geq n_{0} \Rightarrow f(n) < f'(n) \ \forall n \geq n_{0}'$
\end{proof}

\begin{lemma}
    If  $ \lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = r \in (1,\infty) $ , then $Alg1$ will terminate slower than $Alg2$ for any input size $n \geq n_{0}$, with the possibility of computing $n_{0} \in \mathcal{N}^{*}$.
\end{lemma}
\begin{proof}
    $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} > 1 \Rightarrow g_{1}(n) > g'_{1}(n)\ \forall n \geq n_{0} \Rightarrow f(n) > f'(n) \ \forall n \geq n_{0}'$
\end{proof}

\begin{definition}
    Let $f \in \Theta_{1}(g_{1}(n))$ and $f' \in \Theta_{1}(g'_{1}(n))$. $f$ is part of a bigger class than $f'$ iff $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = \infty$.
\end{definition}
\begin{lemma}
    If  $ \lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = \infty $ , then $Alg1$ will terminate slower than $Alg2$ for any input size $n \geq n_{0}$, with the possibility of computing $n_{0} \in \mathcal{N}^{*}$.
\end{lemma}
\begin{proof}
    $\lim_{n\to\infty} \dfrac{g_{1}(n)}{g'_{1}(n)} = \infty \Rightarrow g_{1}(n) > g'_{1}(n)\ \forall n \geq n_{0} \Rightarrow f(n) > f'(n) \ \forall n \geq n_{0}'$
\end{proof}

\section{Comparing algorithms interval-based performances}
Comparing algorithms asymptotic performances based on Normalized rComplexity was a generic way of comparing two algorithms' asymptotic performances for unbounded large input. However, asymptotic performance is not always relevant for computer programs that have a settled \textit{(or a interval-based approximation, with or without weights on sub-specific intervals)} range of input size in order of solving a specific task.
\\ Consider an application responsible of scheduling a football league agenda for the next competitive season, avoiding conflicts and following specific objectives. This problem can be modeled and solved as a \textit{constraint satisfaction problem}(CSP) with different flavors. An asymptotic performance analyzer would simply pick the lowest complexity function in consideration to asymptotic behavior. However, the application is not designed to run on extremely large input size, as the cardinal of the set of all teams part of a football league is a bounded well-known small integer (\textit{most leagues have between 14 and 20 teams}). Therefore, it may be a wise choice to have another method of comparing different algorithms with respect to finite upper bounded input.

For a fixed unique input size $n_{0}\in \mathcal{N}^{*}$, we can use the following natural comparison:
\begin{lemma}
    If  $ \lim_{n\to n_{0}} \dfrac{g_{1}(n)}{g'_{1}(n)} = \dfrac{g_{1}(n_{0})}{g'_{1}(n_{0})} = r \in [0,1) $ , then $Alg1$ will terminate faster than $Alg2$ for the input size $n_{0}$.
\end{lemma}
Symmetrical:
\begin{lemma}
    If  $ \lim_{n\to n_{0}} \dfrac{g_{1}(n)}{g'_{1}(n)} = \dfrac{g_{1}(n_{0})}{g'_{1}(n_{0})} = r \in (1,\infty) $ , then $Alg2$ will terminate faster than $Alg1$ for the input size $n_{0}$.
\end{lemma}
\begin{remark}
    If  $ \lim_{n\to n_{0}} \dfrac{g_{1}(n)}{g'_{1}(n)} = \dfrac{g_{1}(n_{0})}{g'_{1}(n_{0})} = 1$, rComplexity calculus considers $Alg1$ and $Alg2$ equivalent from a computational cost-based perspective.
\end{remark}

For a bounded interval-based input size $n \in [n_{min}, n_{max}]$, we can use the following comparison based on the metrics determined in the previous sections:

\begin{theorem}
    If $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = r \in [0,1) $ , then $Alg1$ will terminate faster (in average) than $Alg2$ for a randomly distribution of input size $n \in [n_{min}, n_{max}]$, assuming \[T_{1}(n_{min}, n_{max}) = \dfrac{\sum\limits_{n=n_{min}}^{n_{max}} g_{1}(n)}{n_{max} - n_{min} + 1}\] and \[T_{2}(n_{min}, n_{max}) = \dfrac{\sum\limits_{n=n_{min}}^{n_{max}} g'_{1}(n)}{n_{max} - n_{min} + 1}\] where $Alg1$ has a complexity function associated to the normalized rComplexity Big 1-Theta defined by $g_{1}$ and $Alg2$ has a complexity function associated to the normalized rComplexity Big 1-Theta defined by $g'_{1}$.
\end{theorem}

\begin{corollary}
    If $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = r \in (1,\infty) $ , then $Alg2$ will terminate faster (in average) than $Alg1$ for a randomly distribution of input size $n \in [n_{min}, n_{max}]$.
\end{corollary}

\begin{remark}
    If  $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = 1$, rComplexity calculus considers $Alg1$ and $Alg2$ equivalent from a computational cost-based perspective for a randomly distribution of input size $n \in [n_{min}, n_{max}]$.
\end{remark}

\begin{remark}
    If  $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = 1$, a tiebreak can be done by a deep analysis of the less dominant features as described in the Enhanced rComplexity model feature extraction in the next chapter.
\end{remark}

\begin{remark}
    Theorem stands likewise using Mean-Value Theorem(Lagrange), with
    \[  T_{1}(n_{min}, n_{max}) = \dfrac{\int\limits_{n_{min}}^{n_{max}} g_{1}(n) dn}{n_{max} - n_{min}}  \]
    \[  T_{2}(n_{min}, n_{max}) = \dfrac{\int\limits_{n_{min}}^{n_{max}} g'_{1}(n) dn}{n_{max} - n_{min}}  \]
    for a randomly distribution of input size $n \in [n_{min}, n_{max}]$.
\end{remark}

Currently, we assumed that the input size is randomly distributed in a bounded interval. Further information about the probabilistic distribution of input size $n \in [n_{min}, n_{max}]$ can offer more insights to this complexity model and a refined comparison model can be established.

\begin{theorem}
    If $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = r \in [0,1) $ , then $Alg1$ will terminate faster (in average) than $Alg2$ for a probabilistic distribution of input size $n \in [n_{min}, n_{max}]$, assuming

    \[T_{1}(n_{min}, n_{max}) = \sum\limits_{n=0}^{f} p_{n} \cdot g_{1}(n + n_{min})\] and \[T_{2}(n_{min}, n_{max}) = \sum\limits_{n=0}^{f} p_{n} \cdot g'_{1}(n + n_{min})\] where $Alg1$ has a complexity function associated to the normalized rComplexity Big 1-Theta defined by $g_{1}$ and $Alg2$ has a complexity function associated to the normalized rComplexity Big 1-Theta defined by $g'_{1}$, and

    \begin{itemize}
        \item $p_{0}$ is the weight associated with $n_{0} = n_{min}$
        \item $p_{1}$ is the weight associated with $n_{1} = n_{min + 1}$
        \item $p_{f}$ is the weight associated with $n_{f} = n_{max}$
    \end{itemize}
    with $f = max - min + 1$.

\end{theorem}

\begin{corollary}
    If $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = r \in (1,\infty) $ , then $Alg2$ will terminate faster (in average) than $Alg1$ for a probabilistic distribution of input size $n \in [n_{min}, n_{max}]$.
\end{corollary}

\begin{remark}
    If  $ \dfrac{T_{1}(n_{min}, n_{max})}{T_{2}(n_{min}, n_{max})} = 1$, rComplexity calculus considers $Alg1$ and $Alg2$ equivalent from a computational cost-based perspective for probabilistic distribution of input size $n \in [n_{min}, n_{max}]$.
\end{remark}

\begin{remark}
    Theorem stands likewise using Mean-Value Theorem(Lagrange), with :
    \[  T_{1}(n_{min}, n_{max}) =\sum\limits_{k=0}^{f-1} p_{k} \cdot \int\limits_{n_{k}}^{n_{k+1}} g_{1}(n) dn  \]
    \[  T_{2}(n_{min}, n_{max}) =\sum\limits_{k=0}^{f-1} p_{k} \cdot \int\limits_{n_{k}}^{n_{k+1}} g'_{1}(n) dn  \]
    for a probabilistic distribution of input size $n \in [n_{min}, n_{max}]$, where:
    \begin{itemize}
        \item $p_{0}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{0}, n_{1}]$
        \item $p_{1}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{1}, n_{2}]$
        \item $p_{f-1}$ is the weight associated with the probability of the input to be bounded in the interval $[n_{f-1}, n_{f}]$
    \end{itemize}
    and $f = max - min + 1$, and $n_{0} = n_{min}, n_{f} = n_{max}$.

\end{remark}


\section{Line Integrals}\label{sec:LineIntegrals}

We have so far integrated ``over'' intervals, areas, and volumes with
single, double, and triple integrals. We now investigate integration
over or ``along'' a curve---``line integrals''
are really ``curve integrals''.\index{line integral}\index{integration!over a curve}

% % % % % % % % % % % %
% Subsections to include

\input{16-vector-calculus/16-2-1-line-integrals-function}
\input{16-vector-calculus/16-2-2-line-integrals-vector-field}
\input{16-vector-calculus/16-2-3-fundamental-thm-line-integrals}


\documentclass[diss.tex]{subfiles}

\begin{document}
\chapter{Preparation}\label{chap:preparation}

\section{Distributed Systems}

In order to serve arbitrary volumes of read/write requests for data, it is common practice to replicate the data across different machines and allow the requests to be split amongst them \cite{baconOS}. I will use \textit{replica} to refer to a node in a network that is part of a distributed system. By default, it is connected to all other replicas and has some state that it wishes to keep consistent with them, which it does by sending/receiving updates about that state.

One formulation of the popular CAP Theorem \cite{CAP} is that for a distributed system you can have at most two of Consistency, Availability and Partition Tolerance, though this has been criticised as misleading \cite{brewer}, and an alternative statement would be that when a network partition occurs, you can choose to have consistency or availability, but not both \cite{kleppmann}. A network partition is where the system is divided into disjoint subsets such that each subset is connected internally, but disconnected from every other subset. In such an event, a modification to one of the replicas in a subset would mean that all the replicas in different subsets would have stale values, and the system would be inconsistent, unless all the other subsets went down so that only the modified subset was available. This is the consistency-availability trade-off the theorem mentions.

\section{CRDTs}
Conflict-Free Replicated Data Types (CRDTs) are a class of data structure designed to specifically for distributed systems to provide Strong Eventual Consistency (SEC) \cite{crdt}.

The CRDT approach is to be available and sacrifice strong consistency (the system behaves as if no replication is present - a read will always give the value of the most recent write). That is, in a network partition, all parts of the system can function, they just may give different results on a read as they can't communicate updates.

For some CRDT-based system, let $o \in \mathcal{O}$ be an operation that can be applied to the state, and $s_i \in \mathcal{S}$ be a possible state in replica $i$. Also, let $m_i \in \mathcal{M}$ be a set of modifications to the state seen by replica $i$. A modification here is either a set of states or of operations, depending on which of the two classes of CRDT (described below) are used.

%SEC
 Eventually consistent systems have the property that any two replicas with the same set of modifications will eventually reach equivalent state. However, some conflict resolution process might be needed to reach the state.
 Strong Eventual Consistency (SEC) goes a step further to assert that as soon as the set of modifications are the same, the states will be equivalent. This means no conflict resolution process is needed. That is, $\forall $ replicas $i,j.~~m_i = m_j \Rightarrow s_i \equiv s_j$.

 % STATE VS OP
CRDTs fall into two broad categories: state-based and operation(op)-based.
% state
A traditional\footnote{There have also been delta state CRDTs proposed \cite{deltacrdts}, where incremental state changes (deltas) are disseminated, but these won't be discussed in detail here.} state based CRDT has a function \textsc{update}: $\mathcal{O} \times \mathcal{S} \rightarrow \mathcal{S}$
which applies operations locally, then distributes the new state to other replicas. On receipt of the new state, another replica applies the function \textsc{merge}: $\mathcal{S} \times \mathcal{S} \rightarrow \mathcal{S}$  for combining incoming states with the local state. This merge function is associative, commutative and idempotent, so that (assuming the states are distributed properly, a liveness guarantee) all replicas will reach equivalent state which is the composed \textsc{merge} of all received states. Here, $\mathcal{M} = \mathcal{P}(\mathcal{S})$ (the powerset).

% op
The op-based approach ($\mathcal{M} = \mathcal{P}(\mathcal{O})$) is similar, but instead of distributing new states, operations are sent to other replicas. The \textsc{update} function (as above) is split into two; \textsc{atSource} is performed only at the operation's source replica, whereas \textsc{downstream} is performed at all replicas. Both share the signature of \textsc{update}.
% network guarantees
This approach additionally requires operations to be delivered only once (or alternatively be idempotent), as for example sending an \textit{increment counter} operation and the network duplicating it means the counter would be incremented twice at other replicas, but only once at the source, giving inconsistent state.
% causal delivery
Furthermore, if operation $A$ \textit{happens-before} \cite{lamportshappensbefore} operation $B$, $A$ should be delivered before $B$ at all replicas. Usually some messaging middleware provides these exactly-once and causally-ordered delivery guarantees. Finally, for all concurrent operations (those not ordered by \textit{happens-before}), the \textsc{downstream} phase commutes. So, for concurrent operations $o_1$ and $o_2$: $$\textsc{downstream}(o_1, \textsc{downstream}(o_2, s)) \equiv \textsc{downstream}(o_2, \textsc{downstream}(o_1, s))$$

% pros/cons
Hence, there are advantages and disadvantages of each approach. On the one hand, state-based CRDTs require transmitting the entire state frequently. When you have a large state and lots of replicas, this can use a lot of bandwidth. Conversely, op-based CRDTs may save you in bandwidth, but require extra messaging guarantees. As well as this, a state-based CRDT can send one state representing multiple updates at once, whereas in the other case every single operation has to be distributed, so there is certainly a tradeoff based on the state size and rate of updates the system has as to which makes more efficient use of the network.
%%%% DIAGRAM STATE VS OP %%%%

% my choice
For my collaborative editing scenario, in general the size of the document will be much larger than any update to it. So, sending the whole document around the network on every update will be much less efficient than just sending the new operations that need to be performed. However, my editor supports offline editing, so upon reconnecting it may be the case that lots of operations are sent separately and it may have been more efficient just to send the state. Despite this, I decided it was better to optimise for the online case (which I expected to be the more common case, appealing to Amdahl's Law) and hence I chose the op-based variety.

\subsection{Ordered Lists}

I define an ordered list as a set of vertices ordered totally by some binary relation $<_v$. For collaborative editing, I use CRDTs to represent an ordered list $\langle v_1, ..., v_n\rangle$ where vertex $v_i = (a_i, id_i)$ is a tuple of an atom $a_i \in \mathcal{A}$ and an identifier $id_i \in \mathcal{ID}$. The document represented is then the concatenation of atoms projected from the ordered vertices. In my implementation, $\mathcal{A}$ is the set of single characters, though in other work it has been the set of all possible whole lines of characters \cite{logoot}. For two lists $l$ and $l'$, let $l \equiv_l l'$ if and only if the vertices are all the same.

Initially, the state $l = \langle \vdash, \dashv\rangle$, two special, invisible vertices such that $\forall $ other vertices $v$,  $$\textsc{identifier}(\vdash) < \textsc{identifier}(v) ~\wedge~ \textsc{identifier}(v) <\textsc{identifier}(\dashv)$$

The two operations that can be performed are \textsc{AddRight}($v_l$,$a$), which inserts a character $a$ to the right of vertex $v_l$ in the document, and \textsc{Delete}($v$), which removes vertex $v$ from the document.
There are different approaches people have taken to representing an ordered list with a CRDT, and I have implemented two of these.
%
%

%
\subsection{Tombstoning Approach}
One approach, called the Replicated Growable Array (RGA) is described in \cite{rga} and summarised nicely in \cite{shapstudy}. A vertex in this system looks like:
$$ v \equiv (a, (t, rid))$$
 The identifier for a vertex is a pair of a timestamp $t \in \mathbb{N}$ and a globally unique replica identifier $rid \in \mathbb{N}$ (so $\mathcal{ID} = \mathbb{N} \times \mathbb{N}$). Identifiers are ordered first by timestamp, then by \textit{rid}. Each vertex additionally has a boolean property \texttt{deleted}, which is initially false. Let there be projection functions associated with all these properties named similarly.\\
The timestamp is such that if the insertion of one vertex $v$ \textit{happens-before} the insertion of another $v'$, $\textsc{timestamp}(v) < \textsc{timestamp}(v')$.
The document state represented is the sequence of projected atoms $a$ of the vertices whose \texttt{deleted} property is false only. The ordering $<_v$ follows from the usual tuple ordering on the identifiers.

\noindent
Below, Algorithm \ref{alg:rga} shows pseudocode for the two main operations for RGA, as well as an auxiliary \textsc{successor} function:
\begin{algorithm}[H]
\caption{RGA}
\label{alg:rga}
\begin{algorithmic}[1]
\Require $v \in l - \{\dashv\}$
\Function{successor}{$v$} \Comment{Finds the next vertex in the list}
\State $succ \gets \min \left\{ v' ~|~ v' \in l -\{\dashv\} ~\wedge~ v <_v v'  \right\}$
\If{$succ \neq None$}
\State \Return $succ$
\Else
\State \Return $v$
\EndIf
\EndFunction
\end{algorithmic}

\hrule

\begin{algorithmic}[1]
\Require $v_l \in l$
\Function{AddRight}{$v_l$, $a$} \Comment{Inserts \textit{a} on the right of vertex $v_l$}
\BState \textsc{atSource}:
\State $t \gets$ \Call{now}{ } \Comment{Generates a fresh timestamp}
\State $v \gets (a,(t, rid))$ \Comment{This replica's \textit{rid}}
\BState \textsc{downstream}:
\State $left \gets v_l$
\State $right \gets \Call{successor}{v_l}$
\While{$left \neq right \wedge \textsc{identifer}(v) < \textsc{identifier}(right)$}
\State $left,~right \gets right, ~\Call{successor}{right}$
\EndWhile
\State $l \gets \langle ..., left, v, right, ...\rangle$
\EndFunction

\Ensure $v_l \in l ~\wedge~ v \in l ~\wedge~ v_l <_v v ~\wedge~ \forall v'.~ v_l<_v v'<_v v \Rightarrow \textsc{identifier}(v) < \textsc{identifier}(v')$
\end{algorithmic}

\hrule

\begin{algorithmic}[1]
\Require $v \in l$
\Function{Delete}{$v$} \Comment{Marks vertex $v$ as deleted}
\BState \textsc{downstream}:
\State $v.\mathtt{deleted} \gets \mathtt{True}$
\EndFunction
\Ensure $v \in l ~\wedge~ v.\texttt{deleted} = \mathtt{True}$
\end{algorithmic}
\end{algorithm}

The details of the \textsc{now} function are described in the next chapter, but loosely it allows a replica to use a timestamp it hasn't seen before.

When characters in the document are deleted under RGA, their vertices are just marked as deleted (\textit{tombstoned}) and the character stops appearing in the representation, but the vertices persist under-the-hood. The obvious weakness of this approach then, is that the memory used by such a CRDT never decreases with time. So, you could have an empty document in front of you that contains a million vertices all marked as deleted which is obviously not ideal.

\subsection{Variable-size Identifier Approach}
Whereas identifiers in RGA consist of a single timestamp and a replica identifier, a different system called Logoot \cite{logoot} has identifiers which grow in size more quickly, but in doing this vertices are allowed to be properly deleted. In RGA, the identifiers grow in memory approximately with the logarithm of the number of vertices (as this is the number of bits needed to count them). Identifiers in Logoot can in the best case use logarithmic space (as a normal timestamp would), but in the worst could (in my implementation) occupy quadratic space (as shown below).

Identifiers are now a globally unique 3-tuple of a positional identifier ($p \in \mathbb{N^*}$), replica identifier ($rid \in \mathbb{N}$) and timestamp ($t \in \mathbb{N}$). Here, $\mathcal{ID} = \mathbb{N^*} \times \mathbb{N} \times \mathbb{N} $. The positional identifier is a sequence of numbers representing a path in a tree.

%%% DIAGRAM %%%
\begin{figure}[H]
\centering
\begin{tikzpicture}
\def \width{5};
\def \depth{-2};
\draw (0,0) -- node[pos=1.1] {$1/\vdash$} ++(-\width,\depth);
\draw (0,0) -- node[pos=1.1] {$k/\dashv$} ++(\width, \depth);

\draw (0,0) -- node[pos=1.1](3) {3} ++ (-\width / 3,\depth) ;
\draw (0,0) -- node[pos=1.1](5) {5} ++ (2, \depth);
\draw (3) --  node[pos=1.15] {3.1}  ++ (0.5, \depth);
\draw (3) -- node[pos=1.15] {$3.13$} ++ (2.5,\depth);

\draw (5) -- node[pos=1.15]{5.2} ++ (0.5, \depth);

\draw[draw=none] (\width+2, 0) -- node[right, pos=0] {depth 0} ++(0, \depth) -- node[right, pos=0] {depth 1} ++(0, \depth) node[right, pos=1.1] {depth 2};

\end{tikzpicture}
\caption{The tree-like structure of identifiers. Each number in the position represents a location to the right of its parent one level down in the tree. The outer two nodes at the top level are the positions of the start and end vertex, $\vdash$ and $\dashv$.}
\label{fig:tree}
\end{figure}
%
%
%
%
\noindent
In Logoot, the ordering of the vertices $<_v$ is the total ordering on their identifiers $\prec$ \cite{logoot} which is defined as follows.
Let: $$id_1 = (p_1, rid_1, t_1)$$ $$id_2 = (p_2, rid_2, t_2)$$
And, (wlog. assume $n \leq m$)
$$ p_1 = p_1^0.p_1^1...p_1^n$$ $$p_2 = p_2^0.p_2^1...p_2^m$$
Then,
$$ p_1 \prec p_2 \Leftrightarrow \exists j \leq m.~ (\forall i<j.~ p_1^i = p_2^i) \wedge (j=n+1 \vee p_1^j < p_2^j)$$
$$ p_1 = p_2 \Leftrightarrow (n=m)~ \wedge~ \forall i. p_1^i = p_2^i$$
Finally, $$id_1 \prec id_2 \Leftrightarrow (p_1 \prec p_2) \vee ((p_1 = p_2) \wedge (rid_1 < rid_2)) \vee ((p_1 = p_2) \wedge (rid_1 = rid_2) \wedge (t_1 < t_2))$$
\\
My implementation the base-doubling strategy from \cite{logoot} so that there are $2^{i-1} \cdot k$ possible spaces at depth $i$. Also, positions are represented internally as single numbers such that the lower $\log_2(base(1))$ bits is the number at depth 1, the next $\log_2(base(2))$ bits is the number at depth 2 et cetera. Therefore, subtraction of positions (as in line 7 of \textsc{alloc} from Algorithm \ref{alg:logoot}) is just integer subtraction.

In the worst case, one might pick the maximum possible identifier at each depth ($2^{i-1} \cdot k$) when inserting a new vertex, then insert another vertex after it (meaning having to go to the next depth) and repeat this. Then, the number at each depth is double the last, so needs one more bit than the last, and the total size of the position is the sum of consecutive integers which grows quadratically.

\begin{algorithm}[H]
\caption{Pseudocode for the two list operations (\textsc{AddRight} and \textsc{Delete}) and auxiliary functions  in Logoot}
\label{alg:logoot}
\begin{algorithmic}[1]
\Require $v \in l - \{\dashv\}$
\Function{successor}{$v$}  \Comment{Finds the next vertex in the list}
\State $succ \gets \min \left\{ v' ~|~ v' \in l -\{\dashv\} ~\wedge~ v <_v v'  \right\}$
\If{$succ \neq None$}
\State \Return $succ$
\Else
\State \Return $v$
\EndIf
\EndFunction
\end{algorithmic}

\hrule

\begin{algorithmic}[1]
\Function{prefix}{$p$, $depth$} \Comment{}
\State $pCopy \gets [~]$\Comment{The empty position}
\State $d \gets 1$
\While{$d < depth$}
\If{$d < p.length$}
$pCopy \gets pCopy.append(p^d)$
\Else{}
$pCopy \gets pCopy.append(0_{base(d)})$ \Comment{s.t. the binary representation of 0 uses $\log_2(base(d))$ digits}
\EndIf
\State $d \gets d+1$
\EndWhile
\Return $pCopy$
\EndFunction
\end{algorithmic}

\hrule

\begin{algorithmic}[1]
\Function{alloc}{$p_l$}\Comment{Logoot's \textit{boundary} strategy}
\State $p_r \gets $ \Call{Successor}{$p_l$}
\State $depth \gets 0$
\State $interval \gets 0$
\While{$interval < 1$}\Comment{Find a gap to insert in}
\State $depth \gets depth + 1$
\State $interval \gets $ \Call{prefix}{$p_r, depth$} $ - $ \Call{prefix}{$p_l, depth$} $-~1$
\EndWhile

\State $step \gets \min(boundary, interval)$
\State $o\!f\!f\!set \in_R [0, step]$
\State \Return \Call{prefix}{$p_l, depth$} $ +~ o\!f\!f\!set$
\EndFunction
\end{algorithmic}

\hrule

\begin{algorithmic}[1]
\Function{AddRight}{$v_l$, $a$}
\BState \textsc{atSource}
\State $t \gets $ \Call{now}{ } \Comment{Generates a fresh timestamp}
\State $newPosition \gets $\Call{alloc}{$v_l.id.p$}
\State $v \gets (a, (newPosition, rid, t)$
\BState \textsc{downstream}
\State $l \gets l \cup \{v\}$
\EndFunction
\Ensure state $l = \langle ..., v_l, v, v_r, ... \rangle$ such that $v_l <_v v <_v v_r$
\end{algorithmic}

\hrule

\begin{algorithmic}[1]

\Require state $l = \langle ..., v_l, v, v_r, ... \rangle \vee v \notin l$
\Function{Delete}{$v$}
\BState \textsc{downstream}
\If {$v \in l$}
\State $l \gets \langle ..., v_l, v_r, ... \rangle$
\EndIf
\EndFunction
\Ensure $v \notin l$
\end{algorithmic}
\end{algorithm}

The \textsc{alloc} function chooses a position for a new vertex at random in between the provided left position $p_l$ and its successor $p_r$. The parameter $boundary$ is a constant used to cap how far away the position is from $p_l$.
A proposed improvement to this scheme is called LSEQ \cite{lseq}. It provides simple changes to \textsc{alloc} that improves space complexity over plain Logoot. For each replica, a mapping from depth to a randomly generated bit is stored. In \textsc{alloc}, if the bit for the required depth is 0, return \textsc{prefix}$(p_l, depth) + o\!f\!f\!set$, otherwise return \textsc{prefix}$(p_r, depth) - o\!f\!f\!set$.
\begin{figure}[H]
\centering
\begin{tikzpicture}
\def \width{5};
\def \depth{2};
\draw (0,0) --  node[pos=1.1] {$p_l$} ++(-\width, -\depth);

\draw (0,0) -- node[pos=1.1] {$p_r$} ++(\width, -\depth);

\draw [dashed] (0,0) -- + (\width/2 , -\depth);
\draw [dashed] (0,0) -- + (-\width/2 , -\depth);
\end{tikzpicture}
\caption{Dotted lines show the two choices LSEQ offers. Choosing a position close to $p_l$ leaves lots of free spaces on the right, but few on the left. Choosing close to $p_r$ leaves lots on the left, but few on the right. If typing sequentially (inserting always at the end), then to minimize the total depth one should always allocate immediately after $p_l$, so all the spaces are filled before having to go deeper.}
\label{fig:lseqalloc}
\end{figure}
That is, as Figure \ref{fig:lseqalloc} shows, at each depth you randomly choose whether to pick a position close to $p_l$ or to $p_r$, then stick with that choice every time that depth is used. To save memory, a good allocation scheme would allocate positions at the smallest depths possible based on future knowledge of what will be inserted, but this obviously isn't possible, so this scheme chooses randomly in the hope of doing well. I implemented LSEQ to be able to contrast it with RGA.
%
%
%
%
%
%
%
%
%
%

\section{Tor}
\subsection{Onion routing}
The Tor\footnote{https://www.torproject.org} \cite{torpaper} overlay network is designed to provide anonymity to its users online. Its name stems from \textit{The Onion Router}, describing how packets are sent around its network. Data is wrapped up in layers of encryption at the source, then this \textit{onion} (because it has layers) is gradually unwrapped as it hops around the network, until it reaches the destination unencrypted.

When a source $S$ wants to send some data to a destination $D$, it chooses a number of Tor \textit{relays} to send the data through (typically 3). The first is called the \textit{entry node}, the last the \textit{exit node}, and any intermediate nodes \textit{middle nodes}. $S$ contacts the entry node directly, and establishes a shared key with it. Then, it asks the entry node to extend the Tor \textit{circuit} to the chosen middle node, and establishes a shared key with that. The process is repeated to reach the exit node. When the whole circuit is established (and $S$ has 3 separate keys), $S$ encrypts the data with all the keys sequentially, in reverse of acquisition order, and sends the data to the entry node. The entry node decrypts the outer layer and forwards it to the middle node, similarly for the middle node, then the exit node decrypts the final layer and forwards to the destination $D$.

\begin{figure}[H]
\centering
\begin{tikzpicture}
\def \i {1};
\filldraw[fill=lightgray!20] (\i,\i/2) -- ++(-2*\i,0) -- ++(0,-\i) -- ++(2*\i,0) -- ++(0, \i);

\node {Data};
\def \i {2};
\draw (\i,\i/2) -- ++(-2*\i,0) -- ++(0,-\i) -- ++(2*\i,0) -- ++(0, \i);
\node[xshift=-2.5ex, yshift=1ex] at (\i,-\i/2){$key3$};

\def \i {3};
\draw (\i,\i/2) -- ++(-2*\i,0) -- ++(0,-\i) -- ++(2*\i,0) -- ++(0, \i);
\node[xshift=-2.5ex, yshift=1ex] at (\i,-\i/2){$key2$};

\def \i {4};
\draw (\i,\i/2) -- ++(-2*\i,0) -- ++(0,-\i) -- ++(2*\i,0) -- ++(0, \i);
\node[xshift=-2.5ex, yshift=1ex] at (\i,-\i/2){$key1$};
%
%
%
%
%
%
%
%   relays
%	  |
%	  v
%
%
%

\node[draw, circle](S) at (-6,-4) {S};
\node[draw, circle](D) at (6, -4) {D};

\node[above  = 0.8cm of S](s)  {} ;
\node [above  = 0.8cm of D](d)  {} ;

%\draw[-triangle 45, decorate, decoration={snake, amplitude=5, segment length=19,}] (s) -- (d);

\node[draw, rectangle split, rectangle split parts=2, align=center, text height=1.75ex,text depth=.25ex](r1) at (-3,-4){entry node\nodepart{second} $key1$};
\node[draw, rectangle split, rectangle split parts=2, align=center, text height=1.75ex,text depth=.25ex](r2) at (0, -4){middle node\nodepart{second} $key2$};
\node[draw, rectangle split, rectangle split parts=2, align=center,text height=1.75ex,text depth=.25ex](r3) at (3,-4){exit node\nodepart{second} $key3$};

\draw[->, line width=0.5mm] (S) -- (r1);
\draw[->, line width=0.5mm] (r1) -- (r2);
\draw[->, line width=0.5mm] (r2) -- (r3);
\draw[->, line width=0.5mm] (r3) -- (D);

\end{tikzpicture}
\caption{The data is encrypted with keys sequentially starting with the key of the last node in the circuit and going backwards.}
\end{figure}

In this way, only the exit node knows the address of $D$, and only the entry node knows the address of $S$, and $D$ knows only the address of the exit node. So, nobody except $S$ knows the address of both $S$ and $D$ for sure. However, the link between the exit node and $D$ is unencrypted, so for confidentiality, encryption at the transport or application layer is needed.
\subsection{Hidden Services}
A traditional service can be configured to accept incoming connections only over Tor, when it is called a \textit{Hidden Service} (HS) \cite{torrendspec}. A client can connect to a HS by specifying its \textit{onion address}, a 16 character string derived from a hash of a special public key it owns. In this way, a HS can hide its location from clients, improving on the asymmetry of knowledge in traditional Onion Routing, so that both parties are anonymous to each other.

After looking up information about the HS in special directories, the client chooses a random Tor relay as a rendezvous point (RP) and both the client and the HS build a Tor circuit to that. The RP relays end-to-end encrypted messages between the client and the HS. Thus the client and HS know only the address of the RP and not each other, whilst the RP knows nothing about either.

\subsection{Client Authentication}

Since the onion address of a HS is derivable from its public key, a connecting client can verify it is connecting to the expected HS. However, the HS knows nothing about the client. It may be desirable for a HS to verify the clients which can build a circuit to it, so the HS protocol supports a couple of methods of client authentication. The one I made use of is called \textit{stealth} authentication.

Here, the HS essentially creates a different identity for each client that wishes to connect, then passes secrets corresponding to each identity to each client through some other channel. When a client looks up the HS in the directory, it finds the entry corresponding to the identity it was told about, and decrypts the HS information using the secret it was given. So, only allowed clients are allowed to even lookup \textit{how} to connect to an authorizing HS. At the time of writing the protocol only allows 16 separately authorized clients per HS, but I decided that this was sufficient for my needs on this project.

\section{Project Development}

\subsection{Choice of platform}
At the beginning of the project, I had to decide in what form an application using my library would be in. I narrowed it down to three choices:
\begin{itemize}
\item A web application using JavaScript
\item An Android application, using Java and the Android API
\item A desktop application using Python
\end{itemize}

I had a little experience with JavaScript and Python on a previous internship, and obviously Java from the Part I Tripos courses. I first ruled out the Android application, as there would have been significant overhead in learning the Android framework properly, and also in actually developing and testing the app. Debugging appeared to be significantly harder remotely. As I knew I wanted to interact with Tor, I needed some way for my library to interact with a Tor daemon running on the same machine. The Tor project actively supports a library for this called Stem\footnote{https://stem.torproject.org/} in Python, so that was what finalised my decision to design a desktop application.

\subsection{Starting Point}
I installed Tor locally, which exposes a SOCKS5 proxy to send data through and runs a daemon process which implements the correct protocols for using the Tor network. To interact with the Tor process, I used the Stem Python library, which provides convenient API calls to work with hidden services. Any other minor libraries used are described in the next section. Otherwise, the code written was from scratch. I initially proposed to make use of some code for a server by Martin Kleppmann, but having planned out how my clients should act, the differences were large enough that writing my own seemed the better approach.

\subsection{Development Plan}
I decided to iteratively develop the code for my project based on each of my goals stated in the project proposal. That way, I could plan and evaluate each part separately. Moreover, a single-pass \textit{Waterfall}-like approach would have been problematic as it is then difficult to adapt to changing or refined requirements after everything is planned. Also, planning a large project such as this before any code was written would have been very difficult as writing code gives you a good insight into further design.

\end{document}