\chapter{Design}
\label{chapter: design}

The main focus of OmniPlay is to record gestures and use them as an input for controlling computer games. In addition, it is important to simplify the setup process for the therapist, ensuring that he does not require any programming experience or knowledge about the underlying system.\\

The graphical user interface (GUI) allows the therapist to interact with the OmniPlay application, providing him with feedback and feedforward on inputting gestures for the patients. Gesture recognition is done by means of machine learning using support vector machines (SVM).

\section{Description of the application}

OmniPlay uses Microsoft's Kinect 2.0 camera as an input device. The principle is that the user can train the program to be able to distinguish multiple postures or gestures from each other. By assigning a virtual keyboard button to each posture or gesture, games can be played by performing them in front of the Kinect. In order to set up the application, there are a number of steps to be taken and guidelines to keep in mind.

\subsection{Principle}

When playing a computer game the conventional way, a person can interact with the game by pressing a keyboard key. Pressing the key then results in an action on the screen (see figure \ref{fig: overview_basic_interaction}).\\

\begin{figure}[H]
\begin{center}
\includegraphics[width=11cm]{Concept1.png}
\caption{\emph{Overview of the interaction between a person and a computer game under conventional use}}
\label{fig: overview_basic_interaction}
\end{center}
\end{figure}

The OmniPlay application can be seen as an additional element between a person and the keyboard key for controlling a game. In short, OmniPlay allows a person to virtually press a key by performing a gesture chosen by the physical therapist (see figure \ref{fig: overview_application_interaction}). This virtual key press is executed by the OmniPlay application, invoking the appropriate Windows operating system command. This invocation has the same effect as pressing the specified key on a physical keyboard. Henceforth, this action is referred to as pressing a virtual keyboard button.\\

\begin{figure}[H]
\begin{center}
\includegraphics[width=16cm]{Concept2.png}
\caption{\emph{Overview of interaction between a person and a computer game using the developed application}}
\label{fig: overview_application_interaction}
\end{center}
\end{figure}

More in detail, the physical therapist comes up with gestures that fit the needs of a patient. He uses the Kinect camera to interact with the application via a GUI. Next, the therapist uses the application to record what gestures need to be done by the patient. Using all of the recorded gestures, an SVM model is created, which is used to predict which gesture is performed by the patient while playing a game.\\

The therapist assigns a virtual keyboard button to each of the gestures. This means that when the patient mimics one of the therapist's gestures, a virtual keyboard button is pressed, as explained before. To start playing a game, the therapist can choose any game that uses keyboard input or let the patient choose his preferred game. This can be any type of computer game and includes, but is not limited to: browser games, games that need to be downloaded and installed, pre-installed games that come with the operating system,\ldots If OmniPlay is running in the background while a computer game is opened, performing a gesture virtually presses the button that is linked to the gesture. By doing this, the patient can interact with the game.\\

By mapping gestures to virtual keyboard buttons, a vast amount of existing games can be played using gesture-based input. It is however limited to button presses. Pointing with the cursor like when using a mouse is not supported by the OmniPlay application. The reason is that every gesture performed is seen as one single action. The gesture of pointing to a specific spot on the screen could overlap with one of the gestures the therapist wants to use. This results in undesirable behavior and negatively impacts the gaming experience for the patient.

\subsection{Setup}

Before the application can be used, the user has to download and install the necessary drivers in order to use the Kinect 2.0 camera. To connect the Kinect 2.0 camera to a computer, an additional adapter is needed so it can be connected using a USB 3.0 port (see figure \ref{fig: kinect}). The use of the Kinect camera requires a socket. For using the Kinect, it is necessary to install a driver. This comes as part of the \emph{Kinect for Windows SDK 2.0}. Microsoft's download page \cite{KinectSDK} states the system requirements for using the Kinect 2.0 camera. These requirements include: a 64-bit processor, dual-core 3.1 GHz or faster processor, USB 3.0, 4 GB of RAM or more, graphics card that supports DirectX 11, Windows 8 or 8.1, Windows Embedded 8 or Windows 10.\\

\begin{figure}[H]
\begin{center}
\includegraphics[width=8cm]{KinectAdapter.png}
\caption{\emph{The Kinect 2.0 camera connected to the adapter}}
\label{fig: kinect}
\end{center}
\end{figure}

The camera has to be placed horizontally in order to function correctly. The distance between the user and the camera may vary between 1.5 and 4 meter, depending on the height of the user and the size of the used screen (see figure \ref{fig: kinect_setup}). The camera can be tilted upwards or downwards. The chosen angle depends on the height of the camera. Both can be chosen freely by the user, but the user has to ensure that he is fully visible at the position he stands. The GUI allows him to verify this. A possible configuration that works well is to have to Kinect camera pointing straight ahead on a table with a height of approximately 0.8 meter.\\

\begin{figure}[H]
\begin{center}
\includegraphics[width=12cm]{SetupKinect.png}
\caption{\emph{Setup for the Kinect camera}}
\label{fig: kinect_setup}
\end{center}
\end{figure}

%Before starting the application through means of an executable file, it is recommended to put this file in a directory that does not need to change over the course of multiple uses. The reason is that a subdirectory is created in the same directory, containing all data files of saved gestures. As long as this data folder and the executable are in the same directory, all previously recorded gestures are loaded when starting the application. If they get separated, the application still can be used, but it appears as if there are no saved gestures and it creates a new data folder in the directory the executable is stored.\\

\section{Graphical user interface}

A large part of this thesis is focused on the design and implementation of a user-friendly interface that can be controlled via input from the Kinect 3D camera. The reasoning behind this is that the user does not want to run back and forth from the standing position in front of the Kinect to the keyboard. Many interfaces have been designed for use with the Kinect, but they are generally only used to navigate to the game and are therefore made up of standard WIMP elements (window, icon, menu, pointer) such as buttons and scrollbars to select your option. In this section, the process of developing the user interface and the properties of the coded prototype are discussed.\\

The goal in this part of the thesis is to find a new way of interacting with a user interface using the Kinect camera that feels natural to the average user. This despite the claims by Norman \cite{Norman2010} that natural user interfaces (NUI), the term that is often used to describe interfaces controlled with 3D cameras, are not natural because there is not the same feedback as when acting on objects in real life. Standard WIMP elements definitely do not feel natural in a NUI. Pointing and clicking on buttons with a pointer may have already become second nature to modern humans, but it is a different thing when they are large and float in the sky or are fixed to a wall and the user has to slap them or close their fist over them without getting any inherent feedback as Wensveen \cite{Wensveen2004} called it. For this reason, elements such as push buttons, scrollbars and floating windows are avoided. Additional hardware is not desirable and voice control is also not considered as it is an area where plenty of research has been conducted already. Implementing such a system is impossible considering the limited time for this thesis. If possible, we wanted to find new ideas on how to control a NUI.

\subsection{Hierarchical task analysis}

To figure out what steps the therapist needs to perform to use the program, a hierarchical task analysis (HTA) is performed. This is a process in which all necessary tasks and subtasks are put into order to get a sense of the properties and elements that the UI must possess. This started as a very technical plan of the tasks that are needed, but was quickly distilled into a simplified visual plan that can be shown to the user to explain how OmniPlay can be used. This plan can be seen in figure \ref{HTA}.

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=12cm]{figures/HTA_plan.png}
		\caption{\emph{The simplified HTA plan}}
		\label{HTA}
	\end{center}
\end{figure}

\subsection{Experimental brainstorm technique}

A brainstorm session with the targeted user, the physical therapist, would give us fresh ideas on how the user expects to interact with a user interface using his body or voice. The expectation was that two distinct observations would be made: how the user wishes to interact with the user interface and how he expects the program to function.\\

The idea is to prevent the subject from being influenced by our ideas or by obvious solutions. Initially, only the basic concept of OmniPlay and the abilities of the Kinect camera are explained. Then the subject is asked how he would imagine every aspect of the application. A few questions from the script are: What information would you expect to see on the screen? How would you start a recording? What would you want to see during recording?\\

To find out whether this experimental brainstorm technique would yield any viable results, brainstorm sessions with our relatives were conducted first. Two iterations were performed with four subjects, the first test was with two males and the second was with two females, of both genders there was one person around 25 years old and one around 55 years old. Each subject was interviewed completely separately to minimize the influence they had on each others ideas.\\

The sessions were very taxing for the test subjects, as the concept we tried to introduce was too abstract and foreign to them. This causes them to be shy with proposals, even after being ensured that there are no bad ideas. It seemed to give them the feeling that they are not smart enough to make good proposals. As a result they fell back on proposing familiar UI elements or concepts such as push buttons or voice control. Getting locked into a mindset that we were designing games to play with the Kinect was also a minor issue, thereby answering the questions with that in mind, which was not the point. After two iterations with minor changes between them, we decided not to subject the physical therapist to this kind of questioning. Setting up a meeting would waste too much of his time with little gain to us. Worse, it might instill the physical therapist with a feeling of dread for our next meeting which might reduce his level of co-operation.\\

Instead the decision was made to brainstorm with each other and our supervisor to develop a few prototypes that we can show to the physical therapist to see which one seems the most user-friendly. The two major interaction patterns that came out of this brainstorm session are, for the purpose of this thesis, named the hints and mime patterns. In the rest of this chapter, the mention of user interface (UI) elements refers to buttons, scrollbars or any other element that the user can see on the screen. When discussing the different concepts, we refer to a movement or position that is taken by the user to activate a function in the program as a sign.\\

The hints pattern implies that the screen is largely devoid of UI elements and actions are performed when the user does a certain movement or takes a certain position. For example, the user forms a cross with his arms to delete a recording. It is called the hints pattern because it is similar to how a game of Hints is played. This is a game where one player tries to depict a concept or activity without using his voice. The other players have to guess what he is trying to say. In this case, the user is the player who is depicting the concept or activity and the program needs to guess what the user means. There is a slight difference here in that the program already knows which signs the user can choose from. Though not always possible, it is better when a sign activates a function in the program that is similar to the concept or activity that this sign refers to in everyday life, as in the last example where an X is related to delete because it generally means bad, wrong or unwanted in Western culture. However, this might be different for other cultures as Norman \cite{Norman2010} pointed out. The implied lack of feedforward means the user should learn all of the signs at the start of the application, putting a large strain on the user's cognitive ability which is not ideal. One could compare it to the shortcut keys on a keyboard such as CTRL+ALT+DEL. There is no indication on the UI that this key combination performs a special function. Norman \cite{Norman2010} made these complaints and also stated that it is hard for the user to find out what he is doing wrong if he does the sign incorrectly. Alternatively, all possible actions could be displayed on the screen around the user with a depiction of the sign. Though this would put less strain on the user's cognitive abilities, he would still have to focus on these pictures and try to figure out what sign they depict. But what if the sign is a movement? It is impossible to depict it statically, so how can you make this clear without distracting the user with constant moving pictures? A more significant problem might be the fact that positions and movements might be recognized by accident. An undo movement could partly solve this, but you also need a redo button to compensate for any accidental undo-activations. The only real benefits to using the hints pattern is that activating an action can be done from anywhere and it could be faster once the user has mastered the signs. Though this depends on the ease of performing the signs and the delay between a sign and the corresponding action in the program.\\

The mime pattern, as the name suggests, is the act of mimicking an action that is needed to act on everyday objects such as opening a door. But where a mime artist does this without any indication of that object being there, in this case an image of, for example, a door is displayed on the screen. The technical term for such a UI element is a metaphor because the function is similar to what a user would do with the object in real life. The power of this pattern is that the user should immediately recognize the UI element and know how to interact with it because it has the same affordances as the object in real life. To achieve the best effect, the UI elements that are chosen should require signs that the user has done frequently already, so that doing them feels natural and familiar. Linking a sign to a function that is similar to the use of that sign in real life, such as going to a different screen when opening the door, improves the interaction, but it is less important than in the hints pattern. Wensveen \cite{Wensveen2004} already said that for this to work correctly, feedback is crucial. The reaction of the UI element must match the expectations of the user. Otherwise, he will be confused and the flow of the program is interrupted. Disadvantages are that the user cannot activate every action from anywhere and it might be slower because of that. Both Wensveen \cite{Wensveen2004} and Norman \cite{Norman2010} state that inherent feedback is lacking here. The user does not feel the UI element with which he is trying to interact.\\

One should prevent mixing the two patterns because the user expects to interact with everything on the screen when using the mime pattern. If one action is activated using the hints pattern without indication, the user will be confused when he has forgotten the sign. Even if there is an indication, the user will likely try to interact with the indicator of the sign as a button, which will not have the desired effect. A combination with regular WIMP UI elements, such as push buttons or scrollbars, is possible for both patterns, but it combines much better with the mime pattern. This is because the user is already interacting with elements of the UI. Pointing at a specific area in the UI does not fit well in the concept of the hints pattern and detracts from the idea of being able to do every action everywhere. To sum up, in the mime pattern the behavior of the UI elements are simple and obvious but possibly more inefficient, like pulling a handle on a slot machine to spin the slots. The hints pattern is more abstract and complex, but allows more efficient interaction once mastered.

\subsection{Prototypes}

With these patterns in mind, the design of the prototypes focuses more on designing UI elements and methods with which the user can interact. They have to be simple, fast, ergonomic, feel natural and not be taxing to the user physically or cognitively while being practical and not easy to activate accidentally. The regular way of testing a concept UI is by doing paper prototyping, which means that the user is presented with a paper version of the UI. Whenever the user touches a UI element the designer manually changes the paper prototype to reflect the activated action. This kind of testing is not suited for the goal of this project. It is more useful for assessing the placement and shape of UI elements and the general flow of a program. What is needed here is a way to interact with the prototypes in a way similar to how the real program would work, but without having to code the whole program. For that reason, we took two example programs from Microsoft's Kinect SDK and made some adjustments to display the user on-screen, in front of the prototype.\\

When designing a motion controlled UI, it is very important to let the user know what the camera sees of their signs. The manner in which this feedback is given has a great impact on how natural the user's experience with the UI is. There are four viable options:\\

 1. A real image of the person as the camera films it (figure \ref{user visualizations} a).\\
 2. A semi-transparent shadow of the user (figure \ref{user visualizations} b).\\
 3. A puppet that copies the user's movement (figure \ref{user visualizations} c).\\
 4. No body is shown, only images that portray the position and state of the hands (figure \ref{user visualizations} d).\\

A real image of the person is the most identifiable for the user, because it is literally an image of himself. This doesn't mean that it is the best option to use in interacting with the UI. Though it has the advantage that it is clear which movement the user carried out, which is good for recording and replaying gestures.\\

 \begin{figure}[H]
 	\begin{center}
 		\includegraphics[width=7cm, height=7cm]{figures/real_image_example.png}
 		\includegraphics[width=7cm, height=7cm]{figures/schadow_example.png}
 		\includegraphics[width=7cm, height=7cm]{figures/puppet_example.png}
 		\includegraphics[width=7cm, height=7cm]{figures/hand_example.png}
 		\caption{\emph{(top-left) a) UI with a real image of the user, (top-right) b) UI with a semi-transparent shadow, (bottom-left) c) UI with a puppet, (bottom-right) d) UI with only hands visible}}
 		\label{user visualizations}
 	\end{center}
 \end{figure}

The most important issue is that the image is taken from the front, so the image of the screen is facing away. Unless the UI elements are drawn on top of the user's image, it is hard to imagine for the user that he is grabbing a handle or pushing a button on the UI. Placing the Kinect camera behind the user to solve this problem is technically not possible because an important part of the Kinect's person recognition is based on recognizing the facial features. This would also require much space to operate because a distance of at least 1 meter from the screen in front of the user and at least 1.5 meter from the Kinect at the back needs to be maintained. When the UI elements are drawn behind the user, he will also obscure any elements directly behind him. Another issue that became apparent during the user tests was that most people are not comfortable with an image of themselves being displayed on the screen, which detracts from the user's experience. The physical therapist we interviewed, who works in a school for disabled children, also noted that this might cause legal problems with parents who do not want unauthorized footage of their child being stored. A final technical issue is that displaying the real image of the user is very resource intensive and in this implementation it would sometimes cause the program to lag.\\

\newpage

A semi-transparent shadow can solve the legal issue and the discomfort experienced by users. It also solves the problem of the image facing away from the UI, because there are almost no features that indicate which way the image is facing, making it look like the shadow is facing towards the UI. By making the image semi-transparent, any UI element behind the user can still be seen. A disadvantage is that it is harder to identify subtle differences between similar recordings. It is also still very resource intensive. Trying to optimize this is beyond the scope of this thesis.\\

A puppet that copies the movements of the user has similar benefits to the shadow and can be transparent or non-transparent. It is far less resource intensive, but makes it even harder to distinguish similar recordings from each other.\\

Showing no body and only hands can obviously only be used to interact with the UI and not for the recording of a gesture. It removes all distractions from the screen and shows only the information that is essential for the user to interact with the UI. But because of this the interaction might feel less natural to the user.\\

The shadow type of representation has the most advantages so it is chosen to make the most prototypes with. Though the more abstract prototypes use the representation that only shows images of the hands of the user to see whether a more abstract representation really detracts from the user experience.\\

In this thesis, three prototypes are discussed, ranging from obvious UIs using mostly mime pattern elements and metaphors to more abstract UIs where more elements of the hints pattern and standard WIMP concepts are used with less mime pattern elements. The first two prototypes are only the screen in which the user can record one gesture multiple times to provide the data for the SVM (step 2 to 4 in figure \ref{HTA}). They use a shadow image of the user to control the UI. The other prototype is a menu to browse to a recording and replay it while the user only sees images of his hands. There are also other prototypes but only these three are discussed to show the kind of elements that were experimented with. These can be ordered as follows:\\

1. A UI with mostly mime pattern elements and metaphors.\\
2. A UI that combines mime patterns with WIMP elements.\\
3. A UI that combines the hints pattern with WIMP elements.\\

The first prototype (see figure \ref{first prototype}) mostly uses the mime pattern and metaphors. The orange color is used to indicate a possibility for an action. On the right the door on the side says exit and would swing open when the user grabs the orange handle and pulls to the left. On the top of the screen, a collection of recordings can be seen pinned to a scrollbar with orange thumbnails. The arrows on the sides imply that the recordings can be moved. The user would grab the orange indicator bar to browse the recordings. To delete a recording the thumbnail of that specific recording can be grabbed and dragged to the recycle bin to release it there. The user can display the recording by dragging it to the projector on the left side of the screen which would display the recording on the projector screen where the large "1" is displayed. In the middle, a camera with its lens pointed towards the user is displayed, by moving his hand forward on the ``opnemen'' button the user could start a recording.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=11.5cm, height=6.5cm]{figures/prototype_1_1_standard.png}
		\caption{\emph{The first prototype screen using metaphors}}
		\label{first prototype}
	\end{center}
\end{figure}

The second prototype in the list combines mime pattern elements with standard WIMP elements to streamline the process (see figure \ref{standard third prototype}). On the right of the screen is a lever that is activated by grabbing the orange part of the lever and pulling it downwards, causing the user to leave this screen. The record button is a push button that the user activates by moving his hand forward over it. This action activates the record screen (see figure \ref{record third prototype}). The window shows what is going to be recorded. The red one indicates where the program will count down from three until the recording starts, allowing the user some time to get to his starting position. All other UI elements are grayed out to indicate that they are not available to interact with. Once the recording is finished, the program returns to the screen in figure \ref{standard third prototype}. On the left is a scrollbar that contains elements, which represent previous recordings of which they show static representations. On the right of it, there is an orange scroll wheel with which the user can scroll through the scrollbar by simply hovering his hand over it. When the user stops scrolling, the elements are locked into a position (see figure \ref{standard third prototype}). The user can slide the top element into the ``TRASH'' or ``PLAY'' area by hovering over it in that direction to play or delete the element.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=11.5cm, height=6.5cm]{figures/prototype_5_3_standard.png}
		\caption{\emph{The standard screen of the second prototype focusing on WIMP elements}}
		\label{standard third prototype}
	\end{center}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=11.2cm, height=5.5cm]{figures/prototype_6_3_record.png}
		\caption{\emph{The record screen of the second prototype}}
		\label{record third prototype}
	\end{center}
\end{figure}

The third prototype uses the hints pattern. The user has to close his hand over the element with which he wants to interact. This opens a WIMP element, called a context menu, with which the user can interact. The user can only see images of his hands. It starts off with a small tutorial screen (figure \ref{first tutorial last prototype}). When the user closes his hand within the orange box, the context menu opens (figure \ref{second tutorial last prototype}). When the user slides his closed fist towards any of the elements, it indicates that the option is chosen (figure \ref{third tutorial last prototype}). Throughout the prototype the user can interact with the chosen recording as seen in figure \ref{example last prototype}. In this instance, he can choose between playing the recording, deleting it or canceling the current menu.

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=9.5cm, height=5cm]{figures/prototype_7_6_tutorial_1.png}
		\caption{\emph{The first tutorial screen of the third prototype}}
		\label{first tutorial last prototype}
	\end{center}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=9.5cm, height=5cm]{figures/prototype_8_6_tutorial_2.png}
		\caption{\emph{The second tutorial screen of the third prototype}}
		\label{second tutorial last prototype}
	\end{center}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=9.5cm, height=5cm]{figures/prototype_9_6_tutorial_3.png}
		\caption{\emph{The third tutorial screen of the third prototype}}
		\label{third tutorial last prototype}
	\end{center}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=9cm, height=5cm]{figures/prototype_10_6_example.png}
		\caption{\emph{An example of how the user can interact with an element of the third prototype}}
		\label{example last prototype}
	\end{center}
\end{figure}

\subsection{Prototype user test}

The main goal of this user test is to see how the user interacts with the different prototypes and how fast can he do the required signs. Additionally, we verify if they feel natural and which one of the prototypes is favored by the user.\\

For these tests, the Kinect is placed on a table of regular height and the user is asked to stay at a distance of 2 to 4 meter from the camera. The screen with the UI is placed directly behind the Kinect camera.\\

The user is first introduced to the concept and flow of our program through the use of the HTA plan as seen in \ref{HTA}. Then, a small demo program is presented showing the capabilities of the concept and the Kinect to give him an idea of the end goal and the possibilities with this concept. After some more introductory questions, the user is assured that he is never at fault and that we are responsible for any misunderstandings he might have. Before the actual tests, permission is asked to film his proceedings. During the tests, the user is reminded that we like to hear his thoughts on what he sees and thinks.\\

Then, the user is asked to interact with the prototypes using the Kinect. To mimic the functions of the UI, a variation of the Wizard of Oz method \cite{WizardOfOz} is used to change the screens whenever the user performs the required action. Some prototypes still require some explanation, but generally, the user is only assisted when he is completely stuck. At the end of each prototype, an explanation of the intent of each UI element is given. After all the tests, each prototype is gone over again without the Kinect to reflect about the performance of each prototype. The user is asked to select a personal favorite.\\

In total, four candidates were subjected to these tests. One person is a physical therapist and three are relatives or friends. However, this paragraph focuses on the results with the physical therapist supplemented with the results of the others. Thereby, the physical therapist is referred to as \emph{the user} and the others as \emph{the other users}. It is important to mention that the user has previous, albeit limited, experience with Kinect games which sometimes influenced his reaction. For instance, he did not know that he could move his hand forward to mimic pushing a button because in his previous experiences, this was never used. For the first prototype (figure \ref{first prototype}), the user was trying to interact with UI elements that were not intended to be interacted with, he later indicated that he did not see the metaphors as they were intended. The screen was too hectic and he didn't see which ones were buttons and which were not. He also found that some signs required him to reach too far. Most of the other users also made the same remarks about the first prototype. A rope in one of the prototypes on which the user could pull was clear to him, but he had no idea how to interact with the scroll wheel, as seen in the second prototype (figure \ref{standard third prototype}), to scroll through the scrollbar. In one of the prototypes, the screen changed based on where the user was positioned. This caused a great deal of confusion. The same goes for the other users. The lever on the third prototype, as discussed above, to leave the screen was clear but he did not like it, as the metaphor was too foreign to him. The user was quick to pick up the controls of the last two prototypes that only showed the hands, but he later indicated that the representation of the hands only didn't feel right to him and that he much preferred to see all his options on the screen. Other users indicated that they expected the UI to react as if it were a touchscreen. When asked how they preferred to see the themselves on the screen, all users experienced discomfort with the real image of themselves and the therapist made the comment about legal issues with the patient as seen before in the discussion of the user's representation. They all preferred the shadow representation. Remarkable was that all the users chose the third prototype, as seen in figure \ref{standard third prototype} earlier on, as their favorite, mostly due to the simplicity and clearness of it.\\

The conclusions are that the most important characteristics are that the screen needs to be simple and straightforward with all of the options displayed on the screen permanently. Modes dependent on subtle parameters, such as the user's position, need to be avoided. The preferred representation of themselves is the shadow. For the purpose of simplifying the UI, regular WIMP elements are still allowed to be used without interfering with the user experience, though clearer affordances for elements that do not look like any object in real life are needed.\\

\subsection{Description of the interface}
\label{interface description}

The end design looks as seen in figure \ref{regular view end design}. Starting from the left, the user can see a gray window in which a recording can be replayed, this is a typical WIMP element because it does not have any elements to it that can be seen as a metaphor to anything the user is familiar with in real life. Right of this there is the rope, which is a very distinct mime pattern element. Further right is a simplified version of the scrollbar containing the recordings with the currently selected recording highlighted in orange. The user can slide this element to the "DEL" bar to delete the element. The black bordered box at the feet of the puppet represent the ground on which the puppet walks so it does not appear to float in mid-air.\\

Though we had previously established that the shadow was the most natural way of representing a user in the interface, performance issues forced us to resort to using the puppet image. The reason why the puppet is not semi-transparent is because then the joints are more visible than the rest because two semi-transparent images are drawn on top of each other which combine into a less transparent color. This breaks the unison of the puppet's coloring making it seem less human which would be a distraction that detracts from the experience of the user. Another detail of the puppet is that the centerpoint of the spine is fixed in the vertical direction but not in the horizontal direction. This eliminates any dependency on the position of the camera so that, as long as the user is within the field of vision of the Kinect, he can always reach every UI element on the screen. This measure introduces two  problems. The biggest problem is that when the user tries to squat the puppet will lift its feet instead of lowering its torso, this might cause confusion and does not feel natural. The other problem is that the user cannot reach any UI element that is placed where it could only be reached by bending downwards. This is not seen as a big problem because such a sign is very taxing for an adult to perform and thus not wise to use in a UI. The puppet is free to move in the horizontal direction but there is an offset on where the puppet is drawn so that the puppet appears to stand between the rope and the scrollbar when the user is standing right in front of the Kinect. In a normal setup the screen will stand behind the Kinect so that the user can look straight ahead when interacting with the UI. The depth coordinates in 3D space are also set to a fixed distance before conversion to a 2D screen to ensure that the puppet is always drawn to the same size no matter how far the user is standing within the field of vision.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=10cm, height=5.5cm]{figures/1_screen_with_user.png}
		\caption{\emph{The regular view of the end design of the UI}}
		\label{regular view end design}
	\end{center}
\end{figure}

An important aspect of the UI is the color coding that is consistent throughout the UI except on the puppet itself though it should be apparent that the puppet is not much more than a representation of the user. This should give the user extra feed forward about what the consequence of his action will be. Orange is indication of interaction possibilities, green means a hand is hovering over the element, blue gives information about what happened or will happen after an action and red indicates a more serious action is about to be activated. The exact meaning of all these color codes will become clear during the description of the UI.\\

Since it is important for the user to know when the system sees that it is grabbing the rope, the UI provides feedback to the user by coloring the hands green when they are open and red when the user makes a fist.\\

In figure \ref{rope handle reaction} a can be seen that when the user hovers his hand over the orange handle of the rope the handle turns green indicating that it can now be grabbed. When the user closes his hand, the handle turns red to show the user that he is pulling it is shown in figure \ref{rope handle reaction} b. The dynamics for pulling the rope are simple, between the starting position and the end position, which is below the starting point, the rope will make the same relative movements in the vertical direction as the user makes with the center of his hand. In this way the interaction is linked in time, speed, direction, dynamics and location of the user's hand. When the rope is near to his end position the record screen is activated. There is no progress bar to indicate how far the user is from activating the function because generally the action of pulling a rope is very sudden and fast, as long as the function is activated before the user runs out of momentum this kind of feedback is not needed here. In the normal state the handle guards are not colored because the affordance is clear enough and a person would not normally grab a handle by the handle guards, they are however colored green and red to make it more apparent to the user that they are acting on it.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=4.5cm, height=6cm]{figures/2_hover_over_rope.png}
		\includegraphics[width=4.5cm, height=6cm]{figures/3_grab_rope.png}
		\caption{\emph{Reaction of the rope handle: a) when hovered over, b) when grabbed}}
		\label{rope handle reaction}
	\end{center}
\end{figure}

The rope activates the record screen seen in figure \ref{first recording screen}. In this screen a separate window is opened with a scaled version of the puppet, another typical WIMP element. The large change in the UI clearly indicates that the user is in a different mode. Here the puppet is also fixed in the horizontal direction to indicate correctly how the program will see the gesture. To avoid confusion the window is opened on top of the last position in which the user was before pulling the rope. Since the goal of this screen is to record one gesture multiple times, two static puppets are drawn behind the user, the light blue puppet shows the starting position and the dark blue puppet shows the end position of the gesture. Care has been taken to choose a color that does not conflict with the green of the user's puppet and doesn't overpower it by being to bright while still being clearly visible and distinguishable. The colors also follow the color code defined earlier in this chapter since it gives information about the first recording. The message "Get Ready" is shown for 1 second before changing to a count down of "3","2","1","Go!". At "Go!" the window's edges turn red to clearly indicate that the program has started recording the user's movements. An example of how this would look can be seen in figure \ref{last recording screen}. The program stops recording when the user remains still for certain amount of time. After that the text changes to "Done!" and after half a second the screen returns to figure  \ref{regular view end design}. The result is that a new element in the scrollbar is added to the top. It is given the next number and set into the orange selection area.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=10cm, height=5.5cm]{figures/4_record_getready.png}
		\caption{\emph{First view of the recording screen}}
		\label{first recording screen}
	\end{center}
\end{figure}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=10cm, height=5.5cm]{figures/6_record_start.png}
		\caption{\emph{View when the program starts recording, between this and the first view there is a countdown from 3, after the recording is started the user must stand still for a certain amount of time to stop, after which the text in the window will read "Done!"}}
		\label{last recording screen}
	\end{center}
\end{figure}

As described before, the scrollbar contains elements that are each linked to a recording of the current gesture. In these boxes a small preview can be seen of the start and end position of the linked recording in the same color scheme as during recording. The numbering of the boxes goes from the first recording as one counting up to the last recording. These numbers are not the same as the IDs of the gesture class as seen further on during the discussion of the implementation of the back end. The order of the numbers is still the same though. When the user hovers one of his hands over the recorded element within the orange selection box the effects as seen in \ref{replay of element six} are triggered.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=10cm, height=5.5cm]{figures/11_replay_recording.png}
		\caption{\emph{The recorded element six is replayed to the user by hovering over it with any hand.}}
		\label{replay of element six}
	\end{center}
\end{figure}

The specific recorded element is replayed within the previously grayed out window on the left. The blue bar on the bottom of the window indicates the progress in playing the recording. As long as the user hovers over the element in the scrollbar, the recording is repeated with half a second delays between repeats. This was a deliberate choice to make sure the user is fully aware of which recording is playing in the window because he should notice that the recording is only playing when he touches it. For the same reasons the number of the recording is also displayed on the left of the screen where it does not obstruct the user in reviewing his recording but is there when the user is uncertain of which recording is playing. Additionally both the orange selection box in the scrollbar and the edges of the display screen are colored green, the color previously linked with hover over information. The redundancy in linking the replay with the recording comes from the fact that the screen is so far from the actual element it is connected to. The scrollbar is on the right because right handed users are more common, so it is easier for them to operate the scrollbar. The delete action is already on the right of it and can hardly be moved to somewhere where it is still as easy to operate. Following both the proximity and similarity law of the Pragnanz law's would make the most sense but placing the screen on the right of the delete button would still have the delete button between it and the recorded element and would place the user's puppet too far from the center of the screen, out of the focus of the user.\\

To activate the scroll function on the scrollbar, the user has to hover his hand within the boundaries of the black bordered box but not within the orange selection box, the black borders of the scrollbar turn green to indicate that the user is hovering over the scrollbar, this is shown in figure \ref{scroll functionality} a. Scrolling only starts when the user has moved a small distance up or down to avoid to scrollbar being immediately stuck to the user which would be annoying. Once scrolling is started it continues as long as the user keeps moving in the same direction, this also counts when going through the selection box area and even when going outside of the scrollbar's borders. This makes sure that the user is not surprised by a sudden stop of the scrolling action when accidentally going inside of the selection box or outside of the scrollbar. The whole reason why the user cannot initiate a scroll within the selection box is because the element that is placed there has to be able to be hovered over to display it, as explained earlier, and it has to be able to be moved to the side for deletion. Accidental activation of the scroll function is minimized by this measure. Though there is more to this, informal tests showed that users who are unfamiliar with the UI often try to scroll down or upward starting from the selection box, the large delay in reaction confuses them. To counter this the activation area of the scroll function is expanded into the orange area for about 15 percent of the selection box height on both the top and the bottom. Once activated the elements stay on a fixed distance from each other but all elements are moved with the same dynamics as used in the record rope, they perform the same vertical relative movement as the hand that is hovering over it. During scrolling all recorded elements are locked to any interactions and the element that will land in the selection box is indicated by a light blue coloring of the white background. The example of scrolling can be seen in figure \ref{scroll functionality} b. When the user stops scrolling the recorded elements are locked into positions as seen in \ref{scroll functionality} a. Only the element in the selection box can be interacted with.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=4.5cm, height=6cm]{figures/9_hover_scrollbar.png}
		\includegraphics[width=4.5cm, height=6cm]{figures/10_scroll_scrollbar.png}
		\caption{\emph{Reaction of the scrollbar: a) when hovered over, b) when scrolling is activated}}
		\label{scroll functionality}
	\end{center}
\end{figure}

The delete sequence of the recorded element in the selection box can be started by hovering over the middle of it. After this the element will follow the relative horizontal movement of the interacting hand between start point and end point with the same dynamics as the scrollbar and the rope. As seen in figure \ref{delete functionality} a, The "DEL" box is filled up with a red progress bar to indicate how far the user still has to move until the item is deleted. When the end point is reached the recorded element turns red and fades away within a halve second to clearly indicate that this recording is deleted. Once the endpoint is reached the action is irreversible and for the duration of the fading the scrollbar and all the elements are locked for any further interaction. An example of this is shown in figure \ref{delete functionality} b. The reason why this action is only started from the middle is to increase the difference between initiating the delete sequence and replaying the recording and also heighten the threshold to start such a serious action as deleting an item. After deleting an element the other recorded elements are renumbered.\\

An important requirement in HCI design is that it should be possible to undo actions. Due to the technical complexity of an undo function and the limited time for this thesis, such a function is not included.\\

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=4.5cm, height=6cm]{figures/12_delete_progress.png}
		\includegraphics[width=4.5cm, height=6cm]{figures/13_delete_complete.png}
		\caption{\emph{The delete sequence: a) user in the process of deleting, the red progress bar indicates how far until deletion the user has to go, b) the endpoint of the action is reached, the item turns red, fades out and is deleted}}
		\label{delete functionality}
	\end{center}
\end{figure}

\section{Gesture recognition}

While the main focus of this thesis is on the user interaction with the OmniPlay application, it is necessary to have a basic algorithm that makes it possible to recognize gestures.\\

In order to provide enough flexibility concerning the type of gestures to be recognized, SVM is used. SVM supports supervised machine learning and its use encompasses two modes: train and predict. After a discussion of the way SVM can be useful for this application, the approach relying on SVM to predict the executed gesture is explained.\\

\subsection{Support vector machines}

A model can be trained with a given data set and labels to classify each gesture. The data set contains all of the recordings of each gesture. A gesture consists of multiple recorded frames. Each frame is classified by using a label. This makes it possible to specify which frames belongs to a certain gesture. Each frame consists of 25 data points, called joints, which are measured by the Kinect camera. A single point is defined in 3D space, meaning that one frame is defined by 75 coordinates. In other words, one frame has 75 features and, as such, one frame can be seen as a single point in a 75-dimensional space.\\

\newpage

It is possible to record the same gesture more than once and include all of these recordings in the data set. All recorded frames of the same gesture receive the same labels to indicate that they are basically the same gesture. While these gestures may appear to be identical, they contain slight variations due to noise during the measurement of a gesture or a slightly varying execution of the gesture by the user.\\

After performing all required gestures, an SVM model is created using an available implementation \cite{LibSVM}. Given a single frame of a gesture, this model can predict which of the recorded gestures the frame belongs to. SVM does this by finding the class of frames that are the most similar to the given frame and returns the label of that class as a result. This also implies the necessity of having multiple trainings recorded for each gesture. Errors in a single training due to noisy measurements of the Kinect camera can lead to wrong predictions. Having multiple trainings minimizes the influence noise has on the prediction and keeps into account that the user executes all gestures with slight variations. As a result, the model can predict more accurately which gesture is performed.\\

However, there are some things to keep in mind when applying this strategy. Firstly, inputting multiple repeats of the same gesture helps creating a better model, which improves predicting the gesture after a model is created. The downside is that it takes more time for the therapist to record all of these gestures. Secondly, when trying to predict a gesture, the generated SVM model always returns the label of the most similar gesture, even if they are not related at all.\\

These problems are tackled as part of the approach to using SVM to predict gestures. Also, the next section discusses how gesture recognition works for the OmniPlay application in more detail.

\subsection{Approach}

Two approaches are considered when it comes down to learning how to recognize gestures. By directly comparing these approaches, it is easier to identify their advantages and disadvantages.\\

The first approach is to add a time stamp to each frame as an extra feature, which indicates the time relative to the first frame of the gesture. This results in having 76 features per frame. The additional time stamp feature can be used to make sure that the gesture is not just similar in space, but also in time. If performing a certain gesture takes $t$ seconds and is captured at a rate of $f$ frames per second, this amounts to 76 $\cdot$ $t \cdot f$ features. For a 4-second gesture recorded at 30 frames per second, which is the highest sampling speed of the Kinect camera, this amounts to 9120 features for a single training of the gesture. In other words, the number of features of one training depends on the duration of the gesture. The entire gesture is classified as a single gesture. During prediction, a gesture with the same number of features can be input. As a result, SVM returns the class of gestures that is the most similar to the given gesture.\\

There are several problems with this first approach. If the number of features of the gesture used for predicting does not match the number of features of the training gestures, the prediction is not accurate and should be discarded. Discarding is necessary as SVM always returns the label of a gesture, even if the predicted gesture is completely unrelated to any of the gestures used for training the model. This poses a problem as different gestures can have a different duration. Even different recordings of the same gesture can take for instance 4 seconds and 4.1 seconds, implying that they have a different number of frames and thus a different number of features. A possible solution is to recalculate the entire gesture and use interpolation to convert the set of frames to a new set with a known, fixed size and preferably with equidistant time stamps. Using time stamps also implies that it is necessary to know when exactly a gesture starts and ends.\\

Furthermore, not just the different recordings of one gesture, but all of the gestures need to have the same number of frames. The gesture to predict is not known beforehand and can only be predicted accurately if the number of features for both the predicted gesture and trained gesture are equal. As a result, it is required that all gestures have an equal amount of features. This means that short gestures need to be mathematically extended with additional frames to match the size of longer gestures. Another side effect is that all gestures need to have the same duration, not just the same amount of features. This limits the therapist in choosing exactly what gestures he wants the patient to perform. Because of this, the action of predicting a gesture can only start when enough time has passed for the player to perform the gesture with the longest duration. More in particular, assume the longest gesture is 5 seconds in length. It then follows that it takes about 5 seconds before prediction can be started. If prediction is started sooner, the gesture has not yet fully been performed by the user. This also means that patients that control the game can only perform one in-game action every 5 seconds. As the games to be played are unknown beforehand, a slow reaction time of the application can render some games unplayable. As such, this approach is not viable.\\

The second approach is to split up one gesture into $n$ smaller gestures and classify each of them differently, assigning a different label to each part of the gesture. Assume that a gesture is split up into 4 smaller parts so that each part contains approximately an equal amount of frames (see figure \ref{fig: gesture_split}). Consider a 4-second gesture sampled at 30 frames per second. The complete gesture consists of 120 frames. By splitting it up into 4 parts, each part contains 30 frames. All first 30 frames are labeled with the same label, for instance 1. The next 30 frames receive a label 2, and so on. In other words, the considered gesture is split up into 4 postures with each having 30 slightly varying trainings. In contrast to the first approach, as described above, prediction does not happen for an entire gesture, but for separate postures. The condition for having executed the entire gesture is that each of the postures are predicted correctly and in the right order. To put it differently, $n$ \emph{pictures} are taken of the gesture and if at some point during prediction all pictures are executed in the right order, the application acknowledges the execution of the gesture.\\

\begin{figure}[H]
\begin{center}
\includegraphics[width=5cm]{SVMGesture.png}
\caption{\emph{An executed gesture, split up into four parts}}
\label{fig: gesture_split}
\end{center}
\end{figure}

The biggest advantage of this approach is the flexibility of it. Each posture corresponds to a single frame, which contains 75 features. This eliminates the need of having gestures with the same length or having to modify training data to have gestures with an equal number of features. It is even possible to not just link a gesture, but also a posture to a virtual keyboard button, which allows the physical therapist to choose the gestures that fit the needs of a patient the best. Additionally, the application reacts immediately to an executed gesture, not after a fixed amount of time. This allows for a wider variety of games being playable using this application.\\

Another advantage is that it solves the issue where SVM always returns the label of a predicted gesture class, even when the patient is not doing anything. A gesture is only recognized when all parts of it are executed. In other words, gestures are not recognized involuntarily and, as such, actions like button presses that influence the game are not executed by accident.\\

A limitation of this second approach is that the physical therapist has no way to choose how fast the patient should perform a gesture. There is no strict requirement for the entire gesture to be executed in about the same time as the recorded gesture used to train the SVM model. If a gesture is executed faster during prediction compared to during training, it is recognized as the same gesture being executed. However, it is possible to solve this timing issue to some extent when the gesture during predicting is performed slower. The gesture can be ignored if more than a certain amount of time passes. This time threshold takes the gesture with the longest duration into account and is chosen higher than this in order to allow all gestures to be executed and successfully recognized.


% JuliaCon proceedings template
\documentclass{juliacon}
\usepackage[ruled,vlined]{algorithm2e}
\setcounter{page}{1}

\keywords{landscape ecology , computational ecology,  julia programming language}

\makeatletter
\newcommand{\removelatexerror}{\let\@latex@error\@gobble}
\makeatother

\begin{document}

\input{header}
\maketitle

\begin{abstract}
Connectivity across landscapes influences a wide range of conservation-relevant ecological processes, including species movements, gene flow, and the spread of wildfire, pests, and diseases. The computational demands of the next generation of connectivity models and the availability of increasingly fine grained remote sensing data drive the need for faster software for connectivity modelling. To address this, we upgraded the widely-used Circuitscape connectivity package to the Julia programming language. The Julia package, Circuitscape.jl, can now solve much larger problems up to an order of magnitude faster, with improved solvers and parallel computing features. We demonstrate scaling up to problems of size 437 million grid cells, with speedups of up to 1800\% over the previous version. These improvements increase the pace of interaction between scientists and key stakeholders, facilitating faster policy decisions.

\end{abstract}

\section{Introduction}
Connectivity models provide important insights into ecological processes that involve variation in movement or flow patterns across heterogeneous environments \cite{crooks2006}. In applied conservation, connectivity maps are incorporated into a wide range of resource evaluations and risk assessments, informing decisions on how to sustain population dynamics and genetic diversity in plant and animal populations \cite{kareiva1995connecting}, how to most effectively prevent infectious disease spread or reduce risks from wildfire \cite{gray2016applying}, and choices of where to invest in land protection or restoration to help support species range shifts under a changing climate \cite{heller2009biodiversity, littlefield2017connecting, keeley2017habitat}.

A common requirement for modeling connectivity is a gridded depiction of the landscape in which values for each cell represent some relative value of “resistance” to movement. These resistance grids are developed through several different methods, often involving iterative processes for categorizing resistance weights for different types of barriers based on expert opinion and information on species’ life histories and movement behaviors \cite{spear2010use, zeller2012estimating}. This grid can then be abstracted as a graph \cite{urban2001landscape}, providing a way to quantify ecological distance measures via graph-theoretic metrics.

The range of mathematical approaches and software tools used for modeling connectivity reflect differences in theoretical approaches, and in the underlying assumptions about how movement proceeds. The classical isolation by distance model (IBD) posits that the least cost distance across the landscape graph acts as a good proxy for ecological distance \cite{wright1943isolation}.  Tools based on this approach typically identify a single “best” route between focal locations, an important result, but one that is highly dependent upon the choice of start and end points, and has limited application if one’s goal is to compare potential options for restoration or protection across a landscape with multiple habitat patches and pathways.  As reviewed in \cite{dickson2019circuit}, from 2006-08, three seminal papers by the late Brad McRae built upon earlier work by \cite{doyle1984random} to demonstrate that isolation by resistance (IBR)\cite{mcrae2006isolation}, which operationalizes the potential for genes and individuals to follow “random walks” across multiple pathways in the same way that electrical current can spread across multiple resistors, can provide an effective tool for considering connectivity potential across landscapes. These insights and McRae’s interest in informing conservation applications inspired the Circuitscape software package \cite{mcrae2008using}, which calculates effective resistance and “current flow,” a measure of net movement probability, across heterogeneous landscapes \cite{dickson2019circuit}.  This approach has evolved into a very flexible set of tools that allow users to vary the resistance grid and identification of what can be connected, enabling modelers to address a wide range of questions related to structural (ability of a landscape to support movement) and functional (modeling that is tailored to species-specific traits) connectivity.
Over the past decade Circuitscape has emerged as the most cited landscape connectivity tool in the world \cite{dickson2019circuit}. In the last year alone, Circuitscape has been cited 129 times, and in the past three years, 480 times. Dickson et al. (2019) reviews 277 applications of the software, in fields ranging from gene flow and animal movement to fire, water, and disease spread, and also describes how outputs are being used to inform conservation decisions.

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{res.pdf}
    \caption{\textbf{An illustration of the impact of coarsening the resistance grid to address computational limitations.}  For this agricultural area along the border of Illinois and Indiana, USA, we show current flow derived from a resistance grid coarsened to 900m (left), compared to a higher-resolution map (180m cells, right).  Both resistance grids were derived from the 2011 NLCD (30m), and run with a “wall-to-wall” application of Circuitscape to evaluate landscape structural connectivity.  All “natural” terrestrial land cover types in the NLCD were assigned low resistance values, with pasture, roads, row crops, etc., assigned higher resistances. The 900m resolution map misses many narrow bands of natural land cover with high current flow (dark blue) indicating high connectivity value along streams and rivers. Analyses and maps by Melissa Clark, to support work in \cite{baldwin2018future}.
}
    \label{fig:my_label}
\end{figure*}

\section{Circuitscape in Julia}
\label{sec:documentclass}
%
The goal of increasing the computational power of Circuitscape has been addressed multiple times by developers Brad McRae and Viral Shah.  McRae’s first version was written in Java, before being ported to MATLAB, to improve ease of development. Then, in collaboration with Tanmay Mohapatra, it was translated to Python for flexible scripting, platform independence and release under an open source license. This package makes heavy use of the numpy \cite{van2011numpy}, scipy \cite{jones2014scipy} and pyamg \cite{bell2015pyamg} numerical libraries and solves problems of up to tens of millions of nodes, but connectivity analysis increasingly finer resolution databases often scaled to hundreds of millions of nodes and dozens of focal node pairs, a workload size well beyond its capacity. Circuitscape needed to be upgraded to support the newer demands of user community.

To address these demands, McRae and Shah collaborated on  a project called GFLOW \cite{leonard2017gflow}, a software tool for conducting circuit-theory based analyses on supercomputers. Written in the C programmming language and making use of solver libraries such as PetSc \cite{balay2004petsc} and BoomerAMG \cite{yang2002boomeramg}, GFLOW yields state of the art performance on large clusters and supercomputers. However, this performance comes at the cost of accessibility (does not work on the Windows operating system, which the majority of Circuitscape users use), composability (ability to integrate with other software libraries) as well as a tangible difficulty in shipping binaries because of their complicated build processes. For many practitioners and researchers, these constraints are a key barrier, as they prevent GFLOW from being supported on the Windows operating system, and limit integration into a decision support tool or workflow with other tools.

As the maintainers of the Circuitscape, we saw the need for an open-source implementation that is easy to maintain, is high performance, is accessible to our user base and works well on every platform. We decided to use the Julia programming language \cite{bezanson2017julia}. Julia is an open source dynamic programming language which combines the readability of scripting languages such as R or Python, with the performance of a statically compiled language such as C or Fortran. With relatively little development effort via a straight reimplementation of the algorithm, Julia allowed us to not only significantly improve the package's computational capacity, but provide new user-facing features for free. For example, Julia's first class sparse matrix library and factorization support allowed us to support multiple solvers. Its modern Just in Time (JIT) compiler enables programmers to write generic code, and then generate specialized code for desired precision, index types and hardware platform. This makes maintenance of the code base simple. Our upgrade, "Circuitscape.jl", is a registered Julia package and is already being used by the community to solve the next generation of connectivity problems.

The rest of this paper is organized as follows: we present an overview of the algorithm in Section 3, before describing the numerical methods and software packages in Section 4. Section 5 then briefly describe the new features introduced in the upgrade and then present benchmarks both on synthetic problems as well as real user data. We then round off this work with a discussion on how speed improvements enable increased collaboration between computer scientists and ecologists enable improved collaboration between ecologists and conservation managers and other stakeholders.

\section{The Algorithm}
\begin{figure}
\removelatexerror
\centering
\begin{algorithm}[H]
\SetAlgoLined
\LinesNumbered
\KwResult{Current (probability) maps $C$, nodal voltages $V$, pairwise resistances}
 Read input raster grid \;
 Read focal nodes \;
Construct undirected, weighted graph connecting neighboring cells\;
Compute graph Laplacian $G$ \;
\For{all pairs of focal nodes}{
    Set up $I$\;
    Solve linear system $GV = I$ \;
    Compute effective resistance $r$\;
    Write nodal voltages $V$ to disk\;
    Compute nodal currents from $I$ and write current map C to disk\
}

 \caption{Circuitscape - Pairwise Mode }
\end{algorithm}
\end{figure}

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{flow.png}
    \caption{\textbf{Stages of computation and inputs/outputs}. The input raster grid to is usually assembled and produced using a GIS software package, and the output current maps are often exported to a GIS software package for postprocessing.}
    \label{fig:flow}
\end{figure*}

Circuitscape takes as input a raster grid, and a list of points which are potential starting and ending points for animal migration. Selecting these points (or focal nodes) on the landscape is often context dependent. For example, they could be two different national parks, or points on the landscape which share important climate properties. The raster grid is a spatial discretization of a heterogeneous landscape where each cell is assigned a value. Each value represents varying qualities of habitat, dispersal routes, or movement barriers. These “resistance” values are often problem specific, and are either empirically derived based on movement or genetic data, or derived via expert opinion. The following content has been summarized in Figure \ref{fig:flow}. This grid can then be represented as a graph, with nodes representing grid cells and edge weights proportional to the movement probabilities or number of migrants exchanged. Other inputs include a list of focal points, which represent points on the landscape of interest to the practitioner. These include habitat patches, protected areas, or populations to connect. These can be specified either through a raster grid with numerical labels for each focal point, or a text file with a list of coordinates. Once the graph is constructed, we then compute its laplacian, which is an alternate representation of the landscape. We then solve for Ohm’s law using the computed graph laplacian as the conductance matrix and the node locations of the focal points as current sources and sinks. On solving the system, we obtain a list of nodal voltages, which are then used to compute branch currents, which represent movement probabilities along branches of the graph. The ecological significance of these various quantities are summarized in related works \cite{mcrae2008using}. The branch currents are then accumulated on each node as nodal currents, which are then written to disk as a raster ASCII grid, for easy import into a geographic information system (GIS).

\section{Numerical Methods}
The vast majority of the computation in Circuitscape is applying Ohm’s law and Kirchoff’s law over very large resistive grids, and solving a large sparse linear system:
$$
GV = I
$$
where $G$ is the graph laplacian representation of the landscape is stored as a large sparse matrix, $I$ are the current sources, and $V$ are the nodal voltages. We provide two solver options to the user: one based on the choleksy factorization and a preconditioned iterative method. The Cholesky factorization \cite{higham2009cholesky} is efficient for applications with smaller study areas and a large number of focal nodes. The sparse matrix is factored once and the solution to multiple pairs is computed via back substitution. Since the cost for backsubstitution is polynomially smaller than the cost of factorization, problems with a large number of focal pairs scale efficiently. The solver that scales to large problems is a preconditioned conjugate gradient (PCG) \cite{trefethen1997numerical}, with an algebraic multigrid preconditioner (AMG) \cite{vanvek1996algebraic}. We leverage Julia implementations of these methods from open source packages IterativeSolvers.jl and AlgebraicMultigrid.jl.

\subsection{Preconditioner Implementation}
Multigrid preconditioners are often used in problems solved across large spatial domains. In general, they take the original grid and generate a hierarchy of coarser grids, via predefined or algebraically derived restriction and interpolation operators. Large problems are solved by restricting quantities down to the coarsest level, obtaining a fast solution, and then interpolating the solution back to the original size. This procedure is often referred to as a “V” cycle. The efficacy of a good multigrid preconditioner is to with obtaining good restriction and interpolation operators. Usually, these operators are derived from the structure of the underlying problem. However, one can also estimate these operators algebraically, from the numerical values at each grid cell. This procedure is often referred to as Algebraic Multigrid (AMG). There are several variants of AMG, which are summarized in \cite{stuben2001review}. We use a variant called the Smoothed Aggregation AMG, which is known to work well for solving matrices generated by elliptic partial differential equations, such as Laplace’s equation. Laplacian matrices are structurally identical to the laplacians of planar graphs that are generated in Circuitscape.

\section{New Features}

\subsection{New Solver Mode}

Circuitscape now supports a new solver based on the Cholesky factorization of the graph laplacian representation of the landscape. We use the SuperNodal sparse cholesky factorization implemented in the CHOLMOD library \cite{chen2008algorithm}. The Julia programming language has first class sparse matrix library and matrix factorization support, which makes it easy to use this solver with a single function call.This solver mode also enables new approaches that divide the landscape into tiles and process them independently \cite{pelletier2014applying}or analyses that move sliding windows across the landscape \cite{mcrae2016conserving}, both of which yield fine scale and localized results. The obvious limitation of this solver, as is of most sparse direct solvers, is that they work very well for relatively small matrices and run out of memory for larger ones. This exponential growth in memory is a by product of the factorization itself, as factors are often less sparse than the original matrix.

\subsection{Generic Programming}

Julia supports the development of generic code while relying on the compiler to generate specialized code for different platforms and precision required. Generic programs are powerful tools for our users, and can be made to adapt to their requirements. For example, the python package had a 32-bit integer type for indexing matrices hardcoded throughout the package, which limited the size of the computation the package ran. This led to crashes when running simulations for hundreds of millions of nodes. One such dataset studying the Mojave desert tortoise \cite{gray2019} used a resistance surface of 437 million pixels. This crashed the old version of Circuitscape, while the upgrade ran smoothly to completion. Circuitscape.jl defaults to a 32-bit integer type for cache efficiency, but provide users with a runtime flag to switch to 64-bit integer to index sparse matrices on the order of hundreds of millions. Since our Julia code was written generically, and this feature came for free without any additional programming effort. Upgrading the python package to support this feature would require significant effort.

\subsection{Improved Parallel Computations}

%Circuitscape.jl introduces support for parallelism on the Windows operating system. This was not possible with the Python package.

Parallel computing in Circuitscape.jl is inherently faster, because the preconditioner can be serialized as a byte stream and sent over the network to other processes. This is not possible in the python code, as the python multiprocessing module internally uses the package pickle to serialize objects, and pyamg objects, implemented in C++, are not “pickle-able” \cite{pydoc}. The ability to serialize native Julia objects significantly reduced the amount of effort to parallelize Circuitscape.
In addition to faster parallel processing, we also extend parallelism support for more problem types in Circuitscape. In the benchmarks section, a dataset which received speed improvements from these new features. The Julia version also lets the user call Circuitscape itself in parallel. As users aim to run Circuitscape over entire countries or continents, landscape are often divided into multiple tiles \cite{pelletier2014applying}. This feature allows users to process (a batch of) these tiles by running different Circuitscape problem instances on each tile in parallel. These parallel features work on all platforms: Linux, MacOS and Windows.

\section{Benchmarks}

\subsection{Experimental Setup}
We conducted our experiments on an Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 384 GB of RAM. We used Julia v1.1.0 and Circuitscape v5.5.0, and compared against Circuitscape v4.0.5

\subsection{Standard Benchmarks}
We benchmarked Circuitscape on standard synthetic problems. These benchmarks represent the most commonly used configuration and mode (Raster Pairwise) but with different problem sizes. These datasets can be found at \url{https://github.com/Circuitscape/BigTests/}. The benchmark results are summarized in Figure \ref{fig:csbench}.

\begin{figure*}
    \centering
    \includegraphics[width=0.5\textwidth]{bench.pdf}
    \caption{\textbf{Results on the standard Circuitscape benchmark suite (smaller is better)}. The red column standards for the Julia package run under the AMG solver and green column stands for the Julia package run with the Cholesky-based CHOLMOD solver. Note that we benchmark only up to size 24M because at higher sizes the CHOLMOD solver runs out of memory and the Python version takes much, much longer. In summary, this chart demonstrates that Julia-CHOLMOD is the right choice for smaller problem sizes and Julia-CGAMG is the right choice for large problem sizes.}
    \label{fig:csbench}
\end{figure*}

\subsection{Benchmarks on Data from Existing Studies}

We ran the Python and Julia version of Circuitscape on a dataset used to model connectivity in the Sonoran desert \cite{drake2017using} (SONORAN). We used to all-to-one scenario \cite{mcrae2009circuitscape}, which models species migrating across the landscape from many different areas to one area. When the researcher used the old version of Circuitscape, this mode did not support parallelism, and the entire computation took over 2 days to run. Parallelism was trivial to support in the Julia version, which reduced execution time to less than 3 hours, resulting in a speedup of nearly 18x. We also benchmarked against a new range-wide model of connectivity for the endangered Mojave desert tortoise \cite{gray2019}(MOJAVE). We found that the Python version crashed, but the Julia package was able to scale effectively. The benchmark results are summarized in Figure \ref{table:bench}.

\begin{figure*}
%\begin{table*}[]
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Benchmark} & \textbf{Size} & \textbf{Number of Solves} & \textbf{Time (Python)} & \textbf{Time (Julia)} \\ \hline
SONORAN            & 3339832       & 6002                      & 56.62 hrs              & 2.838 hrs             \\ \hline
MOJAVE             & 437 million   & 1                         & Crashed                & 3.385 hrs             \\ \hline
\end{tabular}
\centering
%\end{table*}
\caption{\textbf{Benchmarks on data from two conservation studies}. Two metrics determine the execution time of a Circuitscape run: the size of the problem and the number of linear system solves. Our package scales desirably in both directions. }
\label{table:bench}
\end{figure*}

\section{Discussion}

The growth in popularity of circuit theory to model ecological processes has to lead to widespread adoption of the Circuitscape software package. \cite{mcrae_shah_mohapatra_anantharaman}. The Circuitscape project has always evolved with the demands of the users and this upgrade to the Julia programming language seeks to drive the next generation of compute-intensive connectivity models, by shortening execution time from weeks to days and from days to hours. Users no longer have to coarsen their analysis due to computational limitations \cite{drake2017using}. Faster results often enhance stakeholder engagement, improve the feedback loop with policy makers and result in faster turnaround time for conservation decisions. Julia is already being used by ecologists to model ecological networks \cite{timothee_poisot_2018_1438428} and bioenergetic food webs \cite{delmas_eva_2019_2584373} and we see a strong role for it in other areas of computational ecology and conservation science. Its high level mathematical syntax is accessible to scientists while its speed allows them to simulate large, intricate models that capture complexity. We hope to see more ecologists and practitioners use and adopt Julia.

% While connectivity tools typically focus in variation across space, questions related to temporal variability in connectivity, and simulations of how connectivity might change through habitat loss or restoration, would be of great interest as well.

\section{Conclusion}
We present an upgrade to the Circuitscape package, which will allow researchers to analyze ecological processes over large landscapes at fine resolutions. Our upgrade in the high performance Julia programming language presents upto a 1800\% improvement in computation time and the ability to solve landscapes with hundreds of millions (and potentially billions) of grid cells. Julia’s sophisticated compiler allows for faster parallelism, generic programming and composability with other software packages. Circuitscape.jl is open source and is available at \url{https://github.com/Circuitscape/Circuitscape.jl} under the MIT license. Binaries are available on \url{https://circuitscape.org/downloads/}. It can also be installed using the Julia package manager by starting Julia, and entering \texttt{using Pkg; Pkg.add("Circuitscape")}.

\section{Acknowledgement}
The work to update Circuitscape to the Julia computing language was funded through a grant from NASA’s Ecological Forecasting Program (16-ECO4CAST0018), and a grant from the Wilburforce Foundation.  The authors also wish to acknowledge the Wilburforce Foundation’s long-term investment in Circuitscape and related tools through several previous awards to Dr. Brad McRae.  We thank all of the Circuitsape users who have shared datasets, reported issues, and provided feedback on various versions of these tools, and look forward to continued collaboration with the user community.

\input{bib.tex}
%\bibliography{ref}  %%% Remove comment to use the external .bib file (using bibtex).

\end{document}

% Inspired by the International Journal of Computer Applications template


 \documentclass [12pt]{article}

\usepackage {amsmath}
\usepackage {amsthm}
\usepackage {amssymb}
\usepackage {graphicx}
\usepackage {float}
\usepackage {multirow}
\usepackage {xcolor}
\usepackage {algorithmic}
\usepackage [ruled,vlined,commentsnumbered,titlenotnumbered]{algorithm2e} \usepackage {array}
\usepackage {booktabs}
\usepackage {url}
\usepackage {parskip}
\usepackage [margin=1in]{geometry}
\usepackage [T1]{fontenc}
\usepackage {cmbright}
\usepackage [many]{tcolorbox}
\usepackage [colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage {enumitem}
\usepackage {xparse}
\usepackage {verbatim}
\usepackage{listings}
\usepackage{xcolor}
\lstset { %
    language=C++,
    backgroundcolor=\color{black!5}, % set backgroundcolor
    basicstyle=\footnotesize,% basic font setting
}

\DeclareTColorBox {Solution}{}{breakable, title={Solution}} \DeclareTColorBox {Solution*}{}{breakable, title={Solution (provided)}} \DeclareTColorBox {Instruction}{}{boxrule=0pt, boxsep=0pt, left=0.5em, right=0.5em, top=0.5em, bottom=0.5em, arc=0pt, toprule=1pt, bottomrule=1pt} \DeclareDocumentCommand {\Expecting }{+m}{\textbf {[We are expecting:} #1\textbf {]}} \DeclareDocumentCommand {\Points }{m}{\textbf {(#1 pt.)}}

\begin {document}

{\LARGE \textbf {COMP 285 (NC A\&T, Spr `22)}\hfill \textbf {Lecture 4} }
\vspace {1em}
\begin {Instruction}

Adapted From Virginia Williams' lecture notes. Additional credits: J. Su, W. Yang, Gregory Valiant, Mary Wootters, Aviad Rubinstein, Sami Alsheikh.
\end {Instruction}

\begin{centering}
\section*{MergeSort and Introduction to Recurrence Relations}
\end{centering}

\section{MergeSort Cont.}

From last lecture, we will continue our implementation of MergeSort, and go over some of the details regarding correctness and running time.

\subsection{Correctness of MergeSort}
Let's focus on the first question first. As before, we'll proceed by induction. This time, we'll maintain a \textit{recursion invariant} that any time \texttt{MergeSort} returns, it returns a sorted array.

\begin{itemize}
    \item \textbf{Inductive Hypothesis.} Whenever \texttt{MergeSort} returns an array of size $\leq i$, that array is sorted.
    \item \textbf{Base case.} Suppose that $i = 1$. Then whenever \texttt{MergeSort} returns an array of length $0$ or length $1$, that array is sorted. (Since all array of length $0$ and $1$ are sorted). So the Inductive Hypothesis holds for $i = 1$.
    \item \textbf{Inductive step.} We need to show that if \texttt{MergeSort} always returns a sorted array on inputs of length $leq i - 1$, then it always does for length $\leq i$. Suppose that \texttt{MergeSort} has an input of length $i$. Then $L$ and $R$ are both of length $\leq i - 1$, so by induction, $L$ and $R$ are both sorted. Thus, the inductive step boils down to the statement:

    ``When Merge takes as inputs two sorted arrays $L$ and $R$, then it returns a sorted array containing all of the elements of $L$, along with all of the elements of $R$.''

    This statement is intuitively true, although proving it rigorously takes a bit of bookkeeping. In fact, it takes another proof by induction! Check out CLRS Section 2.3.1 for a rigorous treatment.
    \item \textbf{Conclusion.} By induction, the Inductive hypothesis holds for all $i$. In particular, given an array of any length $n$, \texttt{MergeSort} returns a sorted version of that array.
\end{itemize}

\subsection{Running time of MergeSort}
Finally, we get to our first question in this lecture where the answer may not be intuitively obvious. What is the running time of \texttt{MergeSort}? In the next few lectures, we'll see a few principled ways of analyzing the runtime of a recursive algorithm. Here, we'll just go through one of the ways, which is called the \texttt{recursion tree} method.

The idea is to draw a tree representing the computation (see the slides for the visuals). Each node in the tree represents a subproblem, and its children represent the subproblems we need to solve to solve the big sub-problem. The recursion tree for \texttt{MergeSort} looks something like
this:

\begin{figure}[h!]
\centering{}
\includegraphics[scale=0.75]{mergesort_tree.png}
\end{figure}

At the top (zeroth) level is the whole problem, which has size $n$. This gets broken into two sub-problems, each of size $n/2$, and so on. At the $t$'th level, there are $2^t$ problems, each of size $n/2^t$
. This continues until we have $n$ problems of size $1$, which happens at the $log(n)$'th level.

\textbf{Some notes:}

\begin{itemize}
\item In this class, logarithms will \textbf{always} be base $2$, unless otherwise noted.
\item We are being a bit sloppy in the picture above: what if $n$ is not a power of $2$? Then $n/2^j$ might not be an integer. In the pseudocode above, we actually break a problem of size n into problems of size $\lfloor n/2 \rfloor$ and $\lceil n/2 \rceil$. Keeping track of this in our analysis will be messy, and it won't add much, so we will ignore it, and for now we will assume that $n$ is a power of $2$ \footnote{To formally justify the assumption that $n$ is a power of $2$, notice that we can always sort a \textit{longer} list of length $n' = 2^{\lceil log_2(n) \rceil}$. That is, we'll add extra entries, whose values are $\infty$, to the list. Then we sort the new list of length $n'$, and return the first $n$ values. Since $n' \leq 2n$ (why?) this won't affect the asymptotic running time. Also see CLRS Section 4.6.2 for a rigorous analysis of the original algorithm with floors and ceilings.}
\end{itemize}

In order to figure out the total amount of work, we will figure out how much work is done at each node in the tree, and add it all up. To that end, we tally up the work that is done in a particular node in the tree—that is, in a particular call to \texttt{MergeSort}. There are three things:

\begin{enumerate}
    \item Checking the base case
    \item Making recursive calls (but we don't count the work actually done in those recursive calls; that will count in other nodes)
    \item Running Merge.
\end{enumerate}

Let's analyze each of these. Suppose that our input has size $m$ (so that $m = n/2^j$ for some $j$).

\begin{enumerate}
    \item Checking the base case doesn't take much time. For concreteness, let us say that it takes one operation to retrieve the length of $A$, and other operation to compare this length to $1$, for a total of two operations.\footnote{Of course, there's no reason that the ``operation'' of getting the length of $A$ should take the same amount of time as the ``operation'' of comparing two integers. This disconnect is one of the reasons we use big-Oh notation.}
    \item Making the recursive calls should also be fast. If we implemented the pseudocode well, it should also take a constant number of operations.

        \textbf{Aside:} This is a good point to talk about how we interpret pseudocode in this class. Above, we've written \texttt{MergeSort(A[:n/2])} as an example of a recursive call. This makes it clear that we are supposed to recurse on the first half of the list, but it's not clear how we actually implement that. Our ``pseudocode'' above is in fact working Python code, and in Python, this implementation, while clear, is a bit inefficient. That is, written this way, Python will actually copy the first $n/2$ elements of the list before sending them to the recursive call. A much better way would be to instead just pass in pointers to the $0$'th and $n/2 - 1$'st index in the list. This would result in a faster algorithm, but kludgier pseudocode. In this class, we generally will opt for cleaner pseudocode, as long as it does not hurt the asymptotic running time of the algorithm. In this case, our simple-but-slower pseudocode turns out not to affect the asymptotic running time, so we'll stick with this.

    In light of the above Aside, let's suppose that this step takes $m + 2$ operations, $m/2$ to copy each half of the list over, and 2 operations to store the results. Of course, a better implementation of this step would only take a constant number (say, four) operations.
    \item The third thing is the tricky part. We claim that the Merge step also takes about $m$ operations.

    Consider a single call to Merge, where we'll assume the total size of $A$ is $m$ numbers. How long will it take for Merge to execute? To start, there are two initializations for $i$ and $j$. Then, we enter a for loop which will execute $m$ times. Each loop will require one comparison, followed by an assignment to $S$ and an increment of $i$ or $j$. Finally, we'll need to increment the counter in the for loop $k$. If we assume that each operation costs us a certain amount of time, say $Cost_a$ for assignment, $Cost_c$ for comparison, $Cost_i$ for incrementing a counter, then we can express the total time of the Merge subroutine as follows:

    $$
        2Cost_a + m(Cost_a + Cost_c + 2Cost_i)
    $$
    This is a precise, but somewhat unruly expression for the running time. In particular, it seems difficult to keep track of lots of different constants, and it isn't clear which costs will be more or less expensive (especially if we switch programming languages or machine architectures). To simplify our analysis, we choose to assume that there is some global constant $c_{op}$ which represents the cost of an operation. You may think of $c_{op}$ as $\max\{Cost_a, Cost_c , Cost_i, \dots\}$. We can then bound the amount of running time for Merge as
    $$
    2c_{op} + 4c_{op}m = 2 + 4m \text{ operations}
    $$
\end{enumerate}

Thus, the total number of operations is at most
$$
2 + (m+2) + 4m + 2 \leq 11m
$$

using the assumption that $m \geq 1$. This is a very loose bound; for larger $m$ this will be much closer to $5m$ than it is to $11m$. But, as we'll discuss more below, the difference between $5$ and $11$ won't matter too much to us, so much as the linear dependence on $m$.

Now that we understand how much work is going on in one call where the input has size $m$, let's add it all up to obtain a bound on the number of operations required for \texttt{MergeSort}. In a Merge of $m$ numbers, we want to translate this into a bound on the number of operations required for \texttt{MergeSort}. At first glance, the pessimist in you may be concerned that at each level of recursive calls, we're spawning an exponentially increasing number of copies of
\texttt{MergeSort} (because the number of calls at each depth doubles). Dual to this, the optimist in you will notice that at each level, the inputs to the problems are decreasing at an exponential rate (because the input size halves with each recursive call). Today, the optimists win out.

\textbf{Claim 3.} MergeSort \textit{requires at most $11n \log n + 11n$ operations to sort $n$ numbers}.

Before we go about proving this bound, let’s first consider whether this running time bound is good. We covered in last lecture that more obvious methods of sorting, like \texttt{InsertionSort}, required roughly $n^2$ operations. How does $n^2 = n \cdot n$ compare to $n \cdot \log n$? An intuitive definition of $\log n$ is the following: ``Enter n into your calculator. Divide by $2$ until the total is $\leq 1$. The number of times you divided is the logarithm of $n$.'' This number in general will be significantly smaller than $n$. In particular, if $n = 32$, then $\log n = 5$; if $n = 1024$, then $\log n = 10$. Already, to sort arrays of $\approx 103$ numbers, the savings of $n \log n$ as compared to $n^2$ will be orders of magnitude. At larger problem instances of $10^6, 10^9$, etc. the difference will become even more pronounced! $n \log n$ is much closer to growing linearly (with $n$) than it is to growing quadratically (with $n^2$).

One way to argue about the running time of recursive algorithms is to use recurrence relations. A recurrence relation for a running time expresses the time it takes to solve an input of size $n$ in terms of the time required to solve the recursive calls the algorithm makes. In particular, we can write the running time $T(n)$ for \texttt{MergeSort} on an array of n numbers as the following
expression.

\begin{align*}
T(n) &= T(n/2) + T(n/2) + T(Merge(n)) \\
&\leq 2\cdot T(n/2) + 11n
\end{align*}

There are a number of sophisticated and powerful techniques for solving recurrences. We will cover many of these techniques in the coming lectures. Today, we can actually analyze the running time directly.

\textit{Proof of Claim 3} Consider the recursion tree of a call to \texttt{MergeSort} on an array of $n$ numbers. Assume for simplicity that $n$ is a power of $2$. Let’s refer to the initial call as Level $0$, the proceeding recursive calls as Level $1$, and so on, numbering the level of recursion by its depth in the tree. How deep is the tree? At each level, the size of the inputs is divided in half, and there are no recursive calls when the input size is $\leq 1$ element. By our earlier ``definition'', this means the bottom level will be Level $\log n$. Thus, there will be a total of $\log n + 1$ levels.

We can now ask two questions: (1) How many subproblems are there at Level $i$? (2) How large are the individual subproblems at Level $i$? We can observe that at the $i$th level, there will be $2^i$ subproblems, each with inputs of size $n/2^i$.

We’ve already worked out that each sub-problem with input of size $n/2^
i$ takes at most $11n/2^i$ operations. Now we can add this up:

\begin{align*}
    \text{Work at Level }i &= (\text{number of subproblems}) \cdot (\text{work per subproblem}) \\
    &\leq 2^i \cdot 11 \left( \frac{n}{2^i} \right) \\
    &= 11n \text{ operations.}
\end{align*}

Importantly, we can see that the work done at Level $i$ is independent of $i - i$. It only depends on $n$ and is the same for every level. This means we can bound the total running time as follows:

\begin{align*}
    \text{Total number of operations } &= (\text{operations per level}) \cdot (\text{number of levels}) \\
    &\leq (11n)\cdot (\log n + 1) \\
    &= 11n \log n + 11n
\end{align*}
This proves the claim, and we’re done!

\end{document}