% Template for PLoS (Adapted for Frontiers, MdK))
% Version 1.0 January 2009
%
% To compile to pdf, run:
% latex plos.template
% bibtex plos.template
% latex plos.template
% latex plos.template
% dvipdf plos.template

\documentclass[12pt]{article}

% amsmath package, useful for mathematical formulas
\usepackage{amsmath}
% amssymb package, useful for mathematical symbols
\usepackage{amssymb}

% graphicx package, useful for including eps and pdf graphics
% include graphics with the command \includegraphics
\usepackage{graphicx}

% cite package, to clean up citations in the main text. Do not remove.
\usepackage{natbib}
\usepackage{cite}
\usepackage{lineno}
\usepackage{color}

% Use doublespacing - comment out for single spacing
%\usepackage{setspace}
%\doublespacing

% Text layout
\topmargin 0.0cm
\oddsidemargin 0.5cm
\evensidemargin 0.5cm
\textwidth 16cm
\textheight 21cm

% Bold the 'Figure #' in the caption and separate it with a period
% Captions will be left justified
\usepackage[labelfont=bf,labelsep=period,justification=raggedright]{caption}

% Use the Frontiers provided bibtex style

% Remove brackets from numbering in List of References
%\makeatletter
%\renewcommand{\@biblabel}[1]{\quad#1.}
%\makeatother

% Leave date blank
\date{}

\pagestyle{myheadings}
%% ** EDIT HERE **

% the subfigure package
\usepackage{caption}
\usepackage{subcaption}

\DeclareCaptionFormat{subfig}{\figurename~#1#2#3}
\DeclareCaptionSubType*{figure}
\captionsetup[subfigure]{format=subfig,labelsep=colon,labelformat=simple}

% the listings package
\usepackage{listings}
\lstset{
language=C++,
basicstyle=\footnotesize,
numbers=right,
numberstyle=\tiny,
frame=tb,
columns=fullflexible,
showstringspaces=false
}

%% ** EDIT HERE **
%% PLEASE INCLUDE ALL MACROS BELOW

%% END MACROS SECTION

\begin{document}

% Title must be 150 characters or less
\begin{center}
\thispagestyle{empty}
{\Large
\textbf{Parallel MIIND: a Parallel Framework for Populations of Spiking Neurons Using Population Density Techniques}
}
\vspace{1.0cm}
% Insert Author names, affiliations and corresponding author email.
\\
Marc de Kamps$^{1,\ast}$,
David Sichau$^{2}$
\vspace{0.5cm}
\\
\bf{1} School of Computing, University of Leeds, Leeds, West Yorkshire, UK
\\
\bf{2} LAV, Institute of Energy Technology, ETH Zurich, 8092 Zurich, Switzerland
\\
\vspace{0.5cm}
$\ast$ E-mail: m.dekamps@leeds.ac.uk
\end{center}

% Please keep the abstract between 250 and 300 words
\section*{Abstract}

%\section*{Author Summary}
\newpage
\setcounter{page}{1}
\linenumbers

\section{Introduction}
Multiple Interacting Instantiations of Neural Dynamics \citep{dekamps2008} (MIIND) is a simulation framework designed for modeling large networks of
neuronal populations. It is written in C++ and employs MPI \citep{gropp1998} for parallelisation. Python extensions can be provided on request.
Due to its design philosophy it can be easily applied to modeling
biological processes outside computational neuroscience. MIIND abstracts the interaction between neuronal populations into an interaction
between nodes on a directed graph. The neuronal processes describing the activity of the population live on these nodes as objects and evolve
the population’s state. In general we aim to support network processes that use CPU-intensive computations to model nodes, but
require little bandwidth in the internode communication. Neuronal networks where individual spikes are less important than the transmission
of population firing rates are well supported by this paradigm. As such, the processes that are supported by this framework are trivially
parallelizable. Unfortunately, the word trivial does not apply to the actual implementation: we found that in the original code base of MIIND there
were too many obstacles to parallelisation and we decided to start from a fresh code based.

As with any framework, MIIND comes with a number of predefined algorithms, some allowing sophisticated simulations of neuronal
dynamics. The simplest algorithms implement Wilson-Cowan dynamics \citep{wilson1972}, but, uniquely,
MIIND provides an implementation of population density techniques (e.g.) \citep{stein1965,knight1972,knight1996}. They provide efficient simulations
of large populations of neurons by means of a density function over the state variables of a given neuron. Such techniques have undergone rapid
development over the last decade \citep{omurtag2000}. In practice these techniques have been mainly used for leaky-integrate-and-fire (LIF) neurons,
in the so-called diffusion limit: small synaptic efficacies and high firing rate. In  this limit the distribution of membrane potentials of the
population can be modeled by a diffusion process, using Fokker-Planck equations.

Recent progress has extended the usefulness of these technique considerably.  First, the techniques are no longer restricted to the diffusion limit
\citep{omurtag2000,dekamps2003,dekamps2006}, indeed arbitrary large efficacies can be used.  Second, they are no longer restricted to LIF neurons:
a method to model any one-dimensional neuronal model was described recently \citep{dekamps2013}, and will soon be part of the code base.
Others have described density approaches based on 2D neuronal models, including synaptic kinetics, and have established that networks
of 2D populations are computationally still competitive, compared to direct simulation.

The methods described here are at least an order of magnitude more efficient than direct simulation (a real second for a population of LIF neurons
can be modeled in well under 0.5 s simulated time \citep{dekamps2006}). Nevertheless, the computational load is such that a network of hundreds of such
populations would lead to lengthy simulation runs on a serial machine. Therefore investigating the performance of such a network provides a good
benchmark for the scalability of the parallelisation approach.

% Results and Discussion can be combined.
\section{Results}
\subsection{The Framework}
In \citep{dekamps2008} we have outlined the general design ideas of the framework: populations can be modeled by nodes on a directed graph,
which interact via the edges. The evolution of a population is determined by an Algorithm object which lives on the node and which maintains an
AlgorithmGrid. A central simulation loop visits each node in term, collects the contributions from every other node connected to this node and calculates
its input contribution weighted by the link value of the connection. Thus an instantaneous weighted input contribution from the entire network to node will
be calculated, and this will be used by the algorithm to evolve its node state a small moment in time. This will result into an update of the AlgorithmGrid.
By repeating this process for all nodes in the network for a desired period of time, a simulation of the network process is implemented.
The central idea is represented in Figure 1. In the next section we will illustrate how network creation and configuring a simulation is done
from the user's perspective.
\subsection{Simulating Networks as a Use Case}
Here we present a small program that performs a simulation of two neuronal populations: one excitatory and one inhibitory, driven by a common external input.
In listing \ref{code:twopopnet} the shortened code of this simulation is provided.
To increase the readability parameters are not provided.
As a placeholder for the parameters the word PARAM is used.
Additional namespaces are ignored.
For the whole program the reader is refered to the source code where this example is provided as \texttt{two-population.cpp}.
In the next section the setup of an simulation is explained step by step with the two population network.
\begin{lstlisting}[caption=Simulation of a two neuronal population network.,label=code:twopopnet]
int main(int argc, char* argv[]) {
#ifdef ENABLE_MPI
	boost::mpi::environment env(argc, argv);
#endif
	try {
		NodeId id_cortical_background;
		NodeId id_excitatory_main;
		NodeId id_inhibitory_main;
		MPINetwork<OrnsteinUhlenbeckConnection, CircularDistribution> network;

		// Create cortical background, and add to network
		populist::RateFunctor<WeightValue> cortical_background(PARAM);
		id_cortical_background = network.addNode(cortical_background, EXCITATORY);
		// Create excitatory main population
		PopulationAlgorithm_<OrnsteinUhlenbeckConnection> algorithm_excitatory(PARAM);
		id_excitatory_main = network.addNode(algorithm_excitatory, EXCITATORY);
		// Create inhibitory main population
		PopulationAlgorithm_<OrnsteinUhlenbeckConnection>  algorithm_inhibitory(PARAM);
		id_inhibitory_main = network.addNode(algorithm_inhibitory, INHIBITORY);

		// Background and excitatory connection only differ in x, 1- x
		network.makeFirstInputOfSecond(id_cortical_background, id_excitatory_main, PARAM);
		// Excitatory connection to itself
		network.makeFirstInputOfSecond(id_excitatory_main, id_excitatory_main, PARAM);
		// Background connection to I
		network.makeFirstInputOfSecond(id_cortical_background, id_inhibitory_main, PARAM);
		// E to I
		network.makeFirstInputOfSecond(id_excitatory_main, id_inhibitory_main, PARAM);
		// I to E
		network.makeFirstInputOfSecond(id_inhibitory_main, id_excitatory_main, PARAM);
		// I to I
		network.makeFirstInputOfSecond(id_inhibitory_main, id_inhibitory_main, PARAM);

		network.configureSimulation(PARAM);
		network.evolve();
	} catch (std::exception&e) {
		std::cout << e.what() << std::endl;
		return 1;
	}
	return 0;
}
\end{lstlisting}

For parallel MIIND it is important to initialise the mpi environment (see line 3 in listing \ref{code:twopopnet}).
To allow that the code is compilable without an mpi compiler the preprocessor is used to remove the code if mpi is not enabled.
These 3 lines are the only one where actually the user is forced to handle the mpi environment direct.
All other mpi related code is hidden from the user.

After the mpi environment is initialised one can start with setting up an simulation.
The network is created by instantiating a network object (line 9).
The network gets two template parameters.
The first parameter defines the connection type of the network.
In the example above this is a Ornstein Uhlenbeck type of connection.
The second parameter is the distribution of the nodes.
These parameter controls the distribution of the nodes in an mpi environment.
More details will be provided in the next section.

So far only the type of the network is defined.
To define the structure of the network two methods are needed.

The first method \texttt{addNode} adds a new nodes to the network (see line 13, 16 and 19).
This method expects the algorithm executed on this node and the type of the node.
In the example above 3 nodes are generated.
The cortical background driving the simulation with a constant firing rate (line 13).
And an excitatory and inhibitory node where leaky-integrate-and-fire neurons are simulated (line 16 and 19). %what reference for that

The method \texttt{addNode} returns the node id of the generated node.
The node ids start from 0 and are incremented by one for each additional nodes.
The order of the node ids correspond to the chronological order of the calls to \texttt{addNode}.
These ids are needed to identify the individual nodes and are needed for the second method to generate the network structure.

After the creation of the nodes the interactions of the network needs to be created.
Therefore directed connections between two nodes are generated.
Directed Connections between two nodes are generated by calling the method \texttt{makeFirstInputOfSecond}.
The method expects as parameters the node id of the first and the second node in the connection and the parameters of the connection (see line 22 - 32).
In the case of the two population network six connections are generated.
The background node is connected to the exhibitory and the inhibitory node.
The exhibitory and inhibitory nodes are connected to itself and to each other.
With these two methods the structure of the network is constructed.
The structure of the generated network can be seen in figure \ref{fig:twoPopNetwork}.

With these two methods all kind of networks can be generated.
To execute a simulation on that network the simulation needs to be configured.
The configuration is done by providing the simulation parameters to the method \texttt{configureSimulation} (see line 34 in listing \ref{code:twopopnet}).
When the simulation is configured the main simulation loop can be executed with the method \texttt{evolve}.

With the example provided above one can see that the generation of a simulation is straightforward.
It does not need any knowledge about the underlying parallelisation. More details about the parallelisation is provided in the next section.

\subsection{The Simulation Engine: An MPI-enabled Simulation Loop}%TODO(MdK) it is not only the simulation loop which is parallised. Maybe another title would be more suitable
In this section we will describe the mpi-based infrastructure that forms the heart of the framework.

To start with an illustration will be used.
A person wants to solve a large equation consiting of lot of different smaller equations.
In the beginning the person solved the equation on its own.
However it took a large amount of time for one person to solve the equations.
Therefore the management decided to hire additonal people to speed up the process.
To solve the equations faster each person needs to know what equation it should solve and sometimes the extension of another person equation is needed.
To share results and work they need to communicate with each other.
When people communicate with each other they cannot solve equations on the same time.
Additional the they might need lot of extensions of other people so they need to wait until the other people are finished with their calculation.
These two problems show some of the largest issues with parallelisation.
One is the overhead for communication.
And the other the latency.
The illustration above allegorize the message passing interface.

The Message passing interface mpi is a standard to develop applications aimed at large scale parallel applications.
An mpi program resembles an ordinary program written for a serial processor.
This similarity is deceptive, however.
In reallity the code is executed in parallel on different mpi processes.
A mpi process can be either executed on a single core of a multi core processor or on a different CPU.
These execution in different processes results that each process has its own memory space and not a shared memory space like a serial program.
That increases the amount of work to share data between the mpi processes.
For sharing data messages are sent between the individual mpi processes.
To send messages conncetions between the individual processes are needed.
The connections between the individual processes can be either a internal bus or a simple network connection.
However mpi is normally executed on super clusters with a very fast network connection (e.g. InfiniBand).

To speed-up a serial program with mpi the serial program is decomposed in parts that can be executed in parallel.
A serial program that is not embarrasing parallel results in data that needs to be shared between the mpi processes.
Therefore the individual mpi processes needs to communicate with each other to share the data.
To share the data between the mpi processes messages are sent from one mpi process to another with the data.
Compared to computation a message is very slow.
To reduce the overhead of parallelisation it is very important to reduce the number of messages to the minimum.

For the parallelisation of MIIND the idea was to distribute the nodes to MPI processes.
Each node can locally envole one time step without any communication.
The evolve step is responsible for most of the run time of an simulation the run time should be reduced by parallising these method.
To distribute the nodes to the mpi processes a distibution function is used.
As a default distribution function the circular distribution (\texttt{CircularDistribution}) is provided.
The circular distribution assignes the nodes in a circular manar to the mpi processes by using the modulor operator.
For example the network \ref{fig-realNetwork} would be represented internaly like \ref{fig-internalNetwork} when 4 mpi processes are available.
As the default distribution is not always the best choice it can be easily exchanged.
A new distribution can be implemented by implementing the virtual methods of the base class \texttt{NodeDistributionInterface}.
The implementation of a new distribution might be useful if another distribution would decrease the number of messages across the MPI processes (the dotted arrows in figure \ref{fig-internalNetwork}).
To implement a new distribution one might make use of additional knowledge of the network to decrease the messages over MPI process boarders.

With the distribution function the nodes are assigned to distinct mpi processes.
As the nodes are distributed over several mpi processes major changes inside the library where needed.

The parallelisation starts during the creation of the network.
When a node is added to the network the node is only generated on its responsible mpi process.
The method \texttt{addNode} is responsible for creating nodes.

\begin{lstlisting}[caption=The \texttt{addNode} method of the network.,label=code:addNode]
addNode(const AlgorithmInterface<WeightValue>& alg, NodeType nodeType) {
	NodeId tempNodeId = getMaxNodeId();
	if (nodeDistribution.isLocalNode(tempNodeId)) {
		MPINode node = MPINode(...);
		localNodes.insert(std::make_pair(tempNodeId, node));
	}
	incrementMaxNodeId();
	utilities::MPIProxy().barrier();
	return tempNodeId;
}
\end{lstlisting}

In listing \ref{code:addNode} the simplified method is provided.
The method starts with getting the next node id (see line 2 in listing \ref{code:addNode}.
The node id needs to be synchonised between the mpi processes to make sure that each nodes gets an unique id.
The ids of the nodes starts with 0 and increasing numbers are assigned to each node depending on the time of their initialization.
At this moment all mpi processes have the same node id.
To generate a node only on the responsibel mpi process the check on line 3 is carried out.
The \texttt{nodeDistribution} is the choosen distribution class, which checks if the node should be assigned to this mpi process.
If the node should generate on this node the node is generated and stored in the map \texttt{localNodes}, where the nodes on this mpi process are stored with their id as an key.
After the node is generated, or not the node id needs to be increased by calling the method \texttt{incrementMaxNodeId()}.
To avoid side effects the \texttt{addNode} method is synchronised.
For synchronous code mpi barriers are used which make sure that the function is only finished if all processes have executed all code.
As the generation of nodes happen only during the initialization this synchronous code has no impact on the performance.

As the mpi processes have different memory spaces the connections between the nodes cannot be stored anymore by connecting the nodes direct (via pointers or references).
The connections are generated by calls to the \texttt{makeFirstInputOfSecond} method.

\begin{lstlisting}[caption=The simplified \texttt{makeFirstInputOfSecond} method of the network.,label=code:makeFirstInputOfSecond]
makeFirstInputOfSecond(NodeId first, NodeId second, const WeightValue& weight) {
	//Make sure that the node exists and then add the successor
	if (nodeDistribution.isLocalNode(first)) {
		localNodes.find(first)->second.addSuccessor(second);
	}
	// Make sure that the second node exist and then set the precursor
	if (nodeDistribution.isLocalNode(second)) {
		localNodes.find(second)->second.addPrecursor(first, weight, nodeIdsType_[second]);
	}
}
\end{lstlisting}

In listing \ref{code:makeFirstInputOfSecond} the simplified method is provided.
As the connections cannot be stored with pointers or references it is stored by saving the node ids of the precursors respective the successor of a node.
During the \texttt{makeFirstInputOfSecond} method on the local nodes either the successor (line 4) or precursor (line 8) of a node is stored.

After the network has been generated the simulation is initialized.
To avoid to much communication between the nodes each mpi process has its own log and output file.
In the output file the activities of the local nodes are stored.
The advantage of several output files is that the node activities had not to be collected on one mpi process, which would require lot of communication.
This performance gain is paid for by a more difficult post processing.
As the activites of the nodes are distributed into several files and are not stored all in one file.

When the simulation is configured the main simulation loop is started.
The main simulation loop iterates over the local nodes and envoles them one step.
To make sure that all node have recieved and sended their activity at the beginning of each \texttt{evolve} step an barrier is introduced.
At this barrier all nodes waits until all mpi messages are successfully finished.
This barrier increased the performance as the latency of the mpi messages was hidded.
The mpi messages where sent ascynchronus such that the mpi process could continue with other computation and does not need to wait for each to be finished.

The evolve of a node does require that all nodes have the activity of their precursors.
Each node has a temporary storage (\texttt{\_precursorActivity}) of the activity of their precursors.
During the \texttt{evolve} method each node calls his corresponding algorithm and passes the activity of the precursors the weights of the connection and the type of the precursor (see line 3 in listing \ref{code:evolve}).

\begin{lstlisting}[caption=The simplified \texttt{evolve} method of the nodes.,label=code:evolve]
evolve(Time time) {
	while (_pAlgorithm->getCurrentTime() < time) {
		_pAlgorithm->evolveNodeState(_precursorActivity, _weights, time,_precursorTypes);
	}
	this->setActivity(_pAlgorithm->getCurrentRate());
	sendOwnActivity();
	receiveData();
}
\end{lstlisting}

After the algorithm had evolved the state of the node the new rate of the node is stored (line 5).
When the new activity of the node is available the node sends its new activity to all its successors (line 6).
The details of the \texttt{sendOwnActivity} method is explained below.
Additonal the node prepares itself to receive data from his precursors (line 7).
The details of the method \texttt{receiveData} is explained below.
As the  \texttt{sendOwnActivity} and  \texttt{receiveData} are asynchronous they return immediately such that the next node can be envolved without waiting for the data transmission.

The activity of a node is sent to the successors with the help of the  \texttt{sendOwnActivity} method.
In listing \ref{code:sending} the simplified method is provided.
\begin{lstlisting}[caption=The simplified \texttt{sendOwnActivity} method of the nodes.,label=code:sending]
sendOwnActivity() {
  for (auto& it : _successors) {
	if (!_rNodeDistribution.isLocalNode(it)) {
	  MPIProxy().isend(_rNodeDistribution.getResponsibleProcessor(it), _nodeId, _activity);
	}
  }
}
\end{lstlisting}
The method  \texttt{sendOwnActivity}  iterates ofter the ids of the successors of a node (see line 2 in listing \ref{code:sending}.
For each successor it is checked if the successor is on the same mpi process or not (line 3).
If not the activity of the code needs to be send with mpi to the corresponding processor (line 4).
The \texttt{MPIProxy} class is a simple wrapper over the mpi environment to avoid any bare mpi code in library.
\texttt{MPIPoroxy} is explained later into more detail.
The method \texttt{isend} sends an asynchronous mpi message to a processor with some data.
The first parameter \texttt{isend} is the identifier of the mpi process which is responsible for the successor.
The second parameter is a message identifer.
Together with the mpi process these need to be unique.
The last parameter is the actual data that is send to the other node.

After the own data sending was started each node prepares itself to recieve the activities of the other nodes.
Therefore the \texttt{receiveData} method is used.

\begin{lstlisting}[caption=The simplified \texttt{receiveData} method of the nodes.,label=code:receive]
receiveData() {
  int i = 0;
  for (auto it = _precursors.begin(); it != _precursors.end(); it++, i++) {
	if (_rNodeDistribution.isLocalNode(*it)) {
  	  _precursorActivity[i] = _rLocalNodes.find(*it)->second.getActivity();
	} else {
	  MPIProxy().irecv(_rNodeDistribution.getResponsibleProcessor(*it), *it, _precursorActivity[i]);
	}
  }
}
\end{lstlisting}

The \texttt{receiveData} method iterates over all precursors of a node (see line 3 in listing \ref{code:receive}).
For each precursor it is checked if it is local to this mpi process (line 4).
If a node is local the activity is received without any communication with a lookup in the map of the local nodes (line 5).
When a node is not local to this mpi process the method opens an asynchronous receive (line 7).
The first parameter the method \texttt{irecv} is the identifier of the mpi process which is responsible for the precursor.
The second parmeter is the identfier of the message.
Each \texttt{isend} and \texttt{irecv} method must pair depending on responsible processors and the message identifier.
The last parameter is the storage location where the location where the received activity should be stored.
It is the temporary storage of the activities of the precursors which are used during the next \texttt{evolve} step.

These are where all the magic of the parallelisation of MIIND is done.
The algorithms of Parallel MIIND have not changed significantly compared to MIIND.
For an algorithm developer only the way the data is provided has changed.
Additional he does not need to care about concurrency issues as the parallelisation is encapsuled from the algorithm.
This simplifies the development of new algorithms significantly as developers does not need to gain internal knowledge of pmiind.

The most visible change of Parallel MIIND for an application developer is the output.
As explained above each mpi process writes to its own output file.
Instead of one root file with all results for each MPI process a root file with the corresponding local nodes is generated.
Therefore the activity of the nodes are distributed to several files and need to be collected from different files for postprocessing.

To reduce the impact of pmiind all mpi related code is encapsulated in the class \texttt{MPIProxy} and can be turned off.
To improve the readability of the code the mpi related code is wrapped by the class \texttt{MPIProxy}.
Therefore when MPI methods are used boost mpi is not called direct instead the class \texttt{MPIProxy} forwards the calls to boost mpi.
Additional this class stores all open mpi requests of the asynchronous calls.
Reducing the code to handle all open communications as all mpi request are stored in one place and not scattered around the code of the library.
As all mpi related code is encapsulated in one class it can be easily turned off.
When MPI is turned off parallel MIIND can be builded and executed without mpi available on the computer.
This reduces the impact of the development of new algorithms further as the development computer does not need to have mpi installed.
After the testing the parallelisation can be turned on and one can benefit of the speed ups achieved by parallel execution on clusters.

As a parallel environment is very hard to debug a new logging class had been developed.
The class \texttt{Log} provides methods to print logging and debugging messages.
Additional it provides additional information about the mpi environment allowing to identify the mpi process which might have an error.
The generation of a huge number of debug logs is very time consuming. Therefore the logging can be turned off completely for release builds.
Instead off a simple on or off the class provides several log levels allowing a fine granularity of log messages.
This allows the usage of this class not only for debugging purposed but additional for error messages or important information during a simulation run.

\subsection{A Scalable Model of Waves in a Large Network of Leaky-Integrate-and-Fire Neurons}
Consider the local circuit displayed in Figure \ref{fig-network} (A). It consists if an excitatory and inhibitory population, which are fully connected
and driven by a common input. It is possible to predict the firing rate of the populations in the circuit using a number of simple equations
cite{amit199a}, which can be solved with the algorithms provided in MIIND (see section \ref{sec-methods}). A population density-based simulation
shows that the network indeed converges to the predicted rates.

In Figure \ref{fig-network} (C), a hexagonal network is created, whose populations should converge to the same rates. To achieve this, the efficacy
of the local EE connections are halved and each excitatory population is connected laterally with an efficacy that makes up for the reduced
efficacy in the self-connection (section \ref{sec-methods}). Such a network can be easily expanded into an overall hexagonal structure that consists
of a number of rings, and the number of local circuits in the network scales quadratically with the number of rings. By simply expanding the number
of rings, we can investigate the scalability of the network. Finally, a burst from an external input is introduced to the central population.

When no delays are introduced in the lateral connections, the network quickly converges to the same firing rate as for the single local circuit. When
the burst occurs, it is transmitted without delays to the outer populations, which respond immediately. Although this may seem odd, large population
of neurons can respond considerably faster that the time membrane constant of the comprising neurons would suggest. Indeed, population density methods
that model infinitely large populations responds immediately.

When delays are introduced the dynamics of the network becomes very interesting: the network never quite settles in its equilibrium mode as
delayed contributions from further away keep  causing local disturbances (Figure \ref{fig-sim}. The burst is now clearly delayed and
ripples from inside to outside. The outside nodes, however functions as a secondary source, and a rebound on more central populations can
also be discerned (see Figure \ref{fig-sim}).

\section{Discussion}

\section*{Methods}
\label{sec-methods}
\subsection{Network Model}
\subsubsection{Finding Steady State Firing Rate}
\label{sec-steady}
The architecture of the network has been described in Figure \ref{fig-bb}. It consists of a local circuit comprised of an excitatory and an inhibitory
population that are fully connected. We will first describe how this local circuit the local circuit is simulated.
In section \ref{sec-hexagon} we will then describe how this circuit is embedded in an overall hexagonal structure.
Each population
is assumed to be comprised of a large number of spiking LIF neurons. It is assumed that both the populations input and output spike trains
are Poisson distributed and that populations are connected sparsely but by a large number of connections. Under these assumptions an expression
can be found that expresses the population's output firing rate as a function of the input contributions: the so-called gain functions.
Assume a population of LIF neurons with time constant $\tau$, threshold $V_{th}$, reset potential $V_{reset}$ and reversal potential $V_{rev}$
receives Poisson distributed spike trains from another external population, which has a population firing rate of $\nu_{ext}$.
Furthermore, assume that each such input spike raises the membrane potential
instantaneously by a magnitude $h$ (delta peak synapses), that $N_{input}\epsilon$ external neurons connect to a neuron in the population on average,
where $N_{input}$ is the average number of input neurons and $\epsilon$ is a sparseness parameter. The average contribution to the
membrane potential per time unit $\tau$ is then given by:
\begin{equation}
  \mu = \tau \epsilon \nu_{ext} J,
\end{equation}
the variability by:
\begin{equation}
  \sigma^2 = \tau \epsilon \nu_{ext} J^2,
\end{equation}

The population's own firing rate is then given by:
\begin{equation}
\nu = \phi( \mu, \sigma)
\label{eq-transduc}
\end{equation}
with $\phi$ given by:
\begin{equation}
\phi(\mu, \sigma) = \int^{bla}_{ble}du \exp(u^2)(1 + )
\end{equation}
If the population is large, Equation \ref{eq-transduc} is a very good approximation of the firing rate that would be found in a simulation
of LIF neurons indeed. In a network of LIF neuron populations this leads to self-consistency relationships in term of gain functions, that describe
the steady state firing rates that eventually may emerge \footnote{This is not necessarily the case. See section \ref{sec-Fokker-Planck}.} if external contributions to the network are kept constant. For each population $i$ one has:
\begin{equation}
  \nu_i = \phi_i( \mu_i, \sigma_i), i = 0 \cdots N-1
  \label{eq-steady}
\end{equation}
with:
\begin{equation}
  \mu_i = \tau_i (\sum^{N-1}_{j=0} J_{ij}\epsilon_{ij} N_j\nu_j + \sum^{N_{ext}-1}_{k=0} J_{ik} \epsilon_{ik}N_k \nu_k)
  \label{eq-mui}
\end{equation}
\begin{equation}
  \sigma^2_i = \tau (\sum^{N-1}_{j=0} J^2_{ij}\epsilon_{ij} N_j\nu_j + \sum^{N_{ext}-1}_{k=0} J_{ik} \epsilon_{ik}N_k \nu_k)
  \label{eq-sigi}
\end{equation}
The $J_{ij}$ are now the (average) efficacy of population $j$ onto $i$, which can be positive (excitatory) or negative (inhibitory)
with the understanding that  ... Dale's law.

Here $N$ is the number of populations in the network, and the existence of a number of $N_{k}$ external sources is assumed
whose activation drive the dynamics of the network. If a steady state exists where the external populations fire at a constant rate,
then also the network will settle into a steady state where each population $i$ fires with a constant rate. These rates can be found
with Equations \ref{eq-mui} and \ref{eq-sigi}. Although this is an elaborate process, computationally it is much faster than
simulating a network of large populations, waiting until the network relaxates and then finally collating population statistics.

One can find solutions to this set of equations by postulating a state $E_i$ for each node $i$, which evolves according to:
\begin{equation}
  \frac{dE_i}{dt} = -E_i + \phi_i (\mu_i, \sigma_i)
  \label{eq-wc}
\end{equation}
One can pick 'sensible' values  $E_i(0)$ and numerically solve Equation \ref{eq-wc}. After some time, the values $E_i$ will start to approximate
those of the $\nu_i$, given by Equation \ref{eq-steady}.

Although this seems a roundabout way of doing things, in practice it is convenient since the solution of sets of Equations such
as \ref{eq-wc} are precisely what MIIND is designed to do. There is an algorithm provided with MIIND that evolves a node's state
$E_i$ according to Equation \ref{eq-wc}. This allows one to play with the values of connectivities, synaptic efficacies, etc.
and observe their effect on the steady state firing rates of the network.

\subsubsection{Modeling the Network Dynamics}
\label{sec-Fokker-Planck}
The methods described in the last section are well established techniques to find the steady state firing rates
of networks of LIF neuron populations, but there are sitautions where Equation \ref{eq-wc} provides a poor
model for the transient dynamics of the network, or indeed the stability of the steady state solution. It may well be possible
that the network never settles into a steady state. A much better description of the firing rate of neurons is given
by population density methods. Population density methods do not resort to the simulation of large groups of neuronal models,
but instead they define a density function over the neuronal population. Assume the existence of some minimum potential, that
any neuron in the population is likely to cross with vanishing probaility.
One can then define a density function $\rho(V,t)$, defined as the fraction of neurons in the population that have
their membrane potential in interval $[V, v+ dv)$. Clearly:
\begin{equation}
  \int^{V_{th}}_{V_{min}} \rho(V,t) dv = 1
\end{equation}
It can be demonstrated that all relevant quantities that exist at the population level, such as its firing rate, can be calculated
from thsi function. Although mathematically more complex, this method is an order of magnitude more efficient than large
scale neuronal simulations.

Assume the same situation as in the previous section where we defined the gain function of a population, and consider another
population connecting onto this population with all parameters defined in the same way as before.

It can be demonstrated (see \citet{omurtag2000} for a lucid demonstration) that:
\begin{equation}
  \frac{\partial}{\partial t} \rho(V,t) + \frac{1}{\tau}\frac{\partial \rho(V,t)}{\partial V} = \nu_{ext}( \rho(V-h,t) - \rho(V,t)) +\nu(t) \delta(V -V_{reset})
  \label{eq-balance}
\end{equation}
where $\nu(t)$ is given by:
\begin{equation}
  \nu(t) = \int^{V_{th}}_{V_{th} - h} rho(v,t) dv)
\end{equation}
Moreover, one must impose:
\begin{equation}
  \rho(V_{th},t) = 0
\end{equation}
as a boundary condition. Note that in the limit where $h \rightarrow 0$, but $\nu_{ext} h $ is constant, one can Taylor expand the right hand side of
Equation \ref{eq-balance} to obtain the form:
\begin{equation}
\frac{\partial \rho(V,t)}{\partial t} + \frac{1}{\tau}\frac{\partial}{\partial}\left\{(V - \mu)\rho)\right\}  + \frac{sigma^2}{2 \tau}\frac{\partial^2}{\partial v^2} \rho(V,t) = \nu_{ext} \delta (V - V_{reset}),
\label{eq-fp}
\end{equation}
which is a Fokker-Planck equation.
Introducing $\mathcal{L}_{FP}(\mu, \sigma)$ as a shorthand for the second order differential operator defined by Equation \ref{eq-fp}.
The network can now be described by the following set of coupled Fokker-Planck equations:
\begin{equation}
  \mathcal{L}_{FP,i}(\mu_{i},\sigma_{i})\rho_i(V,t) = nu_{i} \delta(V - V_{reset}),
\end{equation}
with:
\begin{equation}
  \nu_i = \int^{V_{th}}_{V_{th}-h} \rho_i(V,t)
\end{equation}
and $\mu_i$, $\sigma_i$ given by Equations \ref{eq-mui} and \ref{eq-sigi}.

Although the state of each individual node is much more complex than for the simulation described in the previous section. The network structure
of the simulation is identical. Again, MIIND provides algorithms that can evolve the state $\rho_i$ of node $i$ numerically, and in this way
an accurate simulation of transient network dynamics and stability can be simulated without having to model thousands of nodes individually, and then
collating their population statistics. The gain in overall efficiency is still large enough to warrant the added complexity of the poulation
density approach.

\subsubsection{A Hexagonal Network of Populations}
\label{sec-hexagon}
Using parameters described as biologically plausible elsewhere \citet{amit1997}, and tinkering with them to obtain
steady state firing rates of 2 Hz for the excitatory population and 5 Hz for the inhibitory population, we arrive at the
connectivity parameters described in Table \ref{tab-con}
\begin{table}[h]
\label{tab-con}
For these parameters the network is stable, although not far away from instability: a mere change of $x$ in $J_{EE}$ brings the network
in a permanent oscillatory state. The convergence of the firing rates and the steady state density distributions are given in Fig \ref{fig-bingo}.
The network can maintain its low level firing rates due a balance of excitation and inhibition and individual neurons in this regime
would fire irregularly.

We now reduce take one such local circuit at the centre of a hexagon, and place six others at the cornes of the hexagon. We
reduce all intra-circuit parameters $J_{EE}$ by a factor half. We connect all excitatory populations laterally by with an efficacy
$\frac{1}{2N_{neigh}}$ where $N_{neigh}$ is the number of neighbours. For the centre node $N_{neigh}= 6$ for the outside nodes $N_{neigh} = 3$.
This ensures that when all excitatory nodes fire with a firing rate of 2 Hz, the neighbour nodes make up for the lost connectivity,
the previous self-stable. Clearly, hexagonal networks with more rings can be created by taking following this procedure, always taking
the correct number of neighbours of a node into account. In general the number of populations in the network $N_{pop}$ is given
by:
\begin{equation}
N_{pop} = 2 ...,
\end{equation}
where the factor of two accounts for there being two populations in the circuit. Finally, it is possible to provide a delay
\caption{Connectivity parameters for the local cortical circuit.}
\end{table}
\section*{Acknowledgments}
This parallelization of MIIND was carried out as a Google Summer of Code 2012 project. We gratefully acknowledge Google's support.

% The bibtex filename
\bibliographystyle{front}
\bibliography{pmiind}

\section*{Figure Legends}
\begin{figure}[tbp]
	\includegraphics[width=0.99\textwidth]{twoPopNetwork.eps}
    \label{fig:twoPopNetwork}
    \caption{The network of the two population network simulation.}
\end{figure}

\begin{figure}[tbp]
	\begin{subfigure}[b]{0.48\linewidth}
		\includegraphics[width=0.99\textwidth]{realNetwork.eps}
        \caption{An example network representation. The circles correspond to the neurons and the arrows between them to directed connection between them.}
		\label{fig-realNetwork}
	\end{subfigure}
	\quad
	\begin{subfigure}[b]{0.48\linewidth}
		\includegraphics[width=0.99\textwidth]{internalNetwork.eps}
        \subcaption{The internal representation of the network. The circles correspond to the neurons. The rounded boxes represent the individual MPI Nodes. For the distribution the default circular distribution was used. The dashed arrows correspond to asynchronous MPI messages. The normal arrows correspond to internal messages.}
		\label{fig-internalNetwork}
	\end{subfigure}
    \caption{The real and internal representation of a small network.}
\end{figure}

\end{document}


\section{Comparison to DNS - Application to Slope Stability Analysis}

To validate the up-scaling methodology used, a simple 2-D slope problem is presented and loaded from the top until failure using both DEM and the up-scaled CDM model. Here, the resultant stress distribution are compared just as failure occurs.

In the DEM model, failure can be assessed based on the unbalanced forces in the model. Since the joints in the model have a stiffness and cohesion, when the slope fails, the explicit quasi-static solution becomes dynamic because of a sudden release of elastic energy and the inability of the applied damping to suppress it all. At this point, the total unbalanced forces in the model increase and the slope can be said to have failed.

For the CDM model, failure can be assessed based on non-convergence of the model when run as an implicit static simulation, which does not converge when the slope fails. The load step in which the CDM model fails to converge because the slope fails dynamically is considered the point of failure.


%% ------------------------------------------------------------------------- %%
\chapter{Introduction and Motivation} \label{Chap:Intro}
\lettrine[findent=2pt]{\textbf{T}}{}he parallel and distributed platforms available today become more and more \textit{heterogeneous}.
Such heterogeneous architectures, composed of several kinds of computing units, have had a growing impact on performance in high-performance computing.
Hardware accelerators, such as General Purpose Graphical Processing Units (in short GPUs), are often used in conjunction with multiple computing units (CPUs) on the same chip sharing the same common memory.
As an instance of this, the number of platforms of the ~\cite{TOP500} equipped with accelerators has significantly increased during the last years. In the future it is expected that the nodes of such platforms will be even more diverse than today: they will be composed of fast computing nodes, hybrid computing nodes mixing general purpose units with accelerators, I/O nodes, nodes specialized in data analytics, etc. In order to use all the computational power available, applications should be composed of multiple tasks that allow the usage of the available resources as efficiently as possible.

The Job Management System (JMS) is the middleware responsible for distributing computing power to applications. A JMS must allocate resources to tasks in order to optimize the use of the available resources while guaranteeing good performance for all applications running in parallel. A promising way to achieve this is using performance prediction. However, in a scenario with millions of processors and a large number of tasks it is very difficult to predict the performance of applications.

Job management systems require that users provide an upper bound of the execution times of their jobs (wall time). Usually, if the execution goes beyond this upper bound, the job is killed. According to~\cite{Gaj:Trystram:2002}, this leads to very bad estimations, with an obvious bias that tends to overestimate this duration. Moreover, those jobs are subject to uncertainties that depend on the context and other running jobs. The uncertain information is generated by communications and executions that depends on the specific context.

The GPUs were initially conceived with the purpose to accelerate the 2D and 3D graphics tasks. However, the evolution of the GPU architectures has gone from a single core to a set of dense arrays of highly parallel cores for more general purpose computation. Not surprisingly, the use of this hardware for parallel computing is a very attractive and it is used in several scientific areas. Moreover, nowadays GPUs can be found in a wide number of electronic devices, just to mention a few: desktops, laptops, smartphones, cars and supercomputers~\citep[chap. 10]{verber2014future}.

GPUs are integrated with multi-core CPUs with the goal to boost more computational processing power. In this way, the regular users have access to a massively parallel environment when acquiring any kind of these devices. Figure \ref{fig:gflops} shows the theoretical performance and bandwidth reached for those devices, this figure shows a comparison between NVIDIA GPUs and current x86 CPUs. The predominant manufacturer of CPUs are Intel and AMD, and of GPUs are NVIDIA and AMD.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{./images/Volta-GPU.png}
\caption{Theoretical performance and bandwidth reached for recent GPUs. Copied from~\cite{CUDAGuide}}
\label{fig:gflops}
\end{figure}

In GPU applications the computation is done with a large amount of data, which expose restrictions on the storage space available in the devices and the latency of data transfers~\citep{Gregg:2011}. GPUs are composed of hundreds and even thousands of simple cores. Figure~\ref{fig:gpuComp} shows a generic scheme of a heterogeneous system composed of a CPU and a GPU. In this figure we can see that CPUs have much more transistors for control and cache, on other hand GPUs have much more ALUs and few transistors for cache and control.

\begin{figure}[htpb]
\centering
\includegraphics[scale=.85]{./images/gpu-computing.png}
\caption{Multiples Cores CPU + Hundreds or Thousands Cores GPU. Copied from~\cite{CUDAGuide}}
\label{fig:gpuComp}
\end{figure}

Last decade with the development of application programming interfaces for GPUs emerged the concept of General Purpose Computing on GPU or GPGPU. GPGPU has evidenced outstanding results in many scientific areas, however, tools to further capitalize these parallel devices and more refined implementations are still emerging~\citep{CUDAGuide,bsgp}. Researchers in this area can create applications with a concept of parallelism that are able to run on both CPU and GPU architectures. Nevertheless, the implementation of an algorithm, for parallel execution on CPU or GPU, does not guarantee that will execute with the same efficiency in the two architectures. In particular, algorithms with parallelism in large blocks of data, i.e. applications with intensity in array arithmetic and linear data structures, can obtain great benefits when they are implemented for execution exclusively on a GPU~\citep{zhong2012data, Benedict:12}.

Nowadays, GPUs are general purpose parallel processing units with accessible programming interfaces, including standard languages such as C, Java, and Python. In particular, the Compute Unified Device Architecture (CUDA) is a parallel computing platform that facilitates the development of applications which execute in GPU~\citep{CUDAGuide}. CUDA was introduced by NVIDIA in 2006 for their GPU hardware line. CUDA is a SIMT programming model, it is an extension of SIMD according to Flynn architecture. MIMD is the model used to characterize a machine in an exchange of messages. CUDA is an extension that developers of different languages can use, there is a phase to preprocess the code and extract the CUDA functions. After, CUDA functions are translated into a pseudo-assembly language named PTX  (Parallel Threads eXecution), and the graphics driver contains a compiler which translates the PTX into a binary code which can be run on the processing cores.

CUDA applications are organized in kernels, which are functions executed in GPUs. Kernels are compounded of thread blocks of execution and each one of these blocks can have hundreds or even thousands of threads. Each kernel is executed asynchronously, and various kernels can execute concurrently. The programming model allows directive of synchronization in different levels of the memory hierarchy.

The software development process and performance modeling of GPU applications require a high degree of understanding and knowledge~\citep{Baghsorkhi:2010:APM}. Performance prediction is often just ignored in a development process, this is due to the in-depth analysis that must be done. Models with specific properties to these problems have been created ~\citep{Skillicorn:1998:MLP} to facilitate programming on parallel machines.  Properties such as synchronization and division of work between computing and communication improve the way developers can solve a problem and how the performance of applications running on parallel machines can be modeled.

The accuracy of a GPU performance model is subject to low-level elements such as instruction pipeline usage and cache hierarchy. GPU performance approaches its peak when the instruction pipeline is saturated but becomes unpredictable when the pipeline is under-utilized~\citep{Baghsorkhi:2010:APM}. Effects of cache hierarchy and memory-access divergence are also critical to GPU performance models. Some parallel programs can be efficiently executed on some architectures, but not on others.

Performance prediction over these devices is a great challenge because hardware characteristics can impact their performance in different ways. There are different approaches to do this, such as analytical modeling and machine learning techniques. Analytic predictive models are useful, but require the manual inclusion of interactions between architecture and software, and may not capture the complex interactions in GPU architectures. Machine learning techniques can learn to capture these interactions without manual intervention but can require large training sets.

There are different models of parallel machines that help to deal with these problems. Among these models, we have the Parallel Random Access Memory (PRAM), Bulk Synchronous Parallel (BSP) and Coarse Grained Multicomputer (CGM). The main objective of a parallel computing model is to provide a set of parameters that have an impact on the performance of applications in parallel architectures, and thus to be able to simulate the behavior of these applications in their respective architectures.

The Bulk Synchronous Parallel is a computing model for parallel computing introduced for~\cite{Valiant:1990}, BSP model provides a simple but accurate characterization of all parallel machines-past, present, and future. There are other parallel models parameterized, almost all of them using or extending the focus of the BSP model. \cite{Dehne:2002} studied the problem to design scalable parallel geometric algorithms for the Coarse Grained Multicomputer model which are optimal or at least efficient for a wide range of the ratio $\frac{n}{p}$, with $n$ the size of the problem and $p$ the number of processors.

The BSP model allows developers to design algorithms and software that can run on any standard parallel architecture with very high performance~\citep{goldchleger2005implementation}. This model may potentially serve as a unified programming model for both coarse-grained and fine-grained. The model BSP has been implemented as API libraries and programming languages \citep{BSPLib} and API libraries have been developed to use BSP on GPU architectures~\citep{bsgp}, these developments are to create scientific applications in massively parallel environments computing in an easy and better way. Recently, ~\cite{Valiant:2011} has implemented the BSP model over multicore architectures. The implementation of BSP model for multicore architecture incorporates the memory size shared as a further parameter in the computer. The BSP model is a good alternative to tackle parallel problems in massively parallel architectures.

%Data are produced either by the Job management systems, to effectively leverage these platforms, and by the user's applications that are executed. These applications could be in execution, awaiting execution, monitoring the machine, so forth. While this happens, a lot of information is registered in traces. There are several recent works on using this information~\citep{Tsafrir:Backfilling:2007,Lublin01theworkload,Ernemann:2003}, in other cases these data are ignored by the job management systems~\citep{andreetto2008glite}.

The developed methods in the area of \textbf{big data} (especially machine learning) can be used for the scheduling of jobs in new computing platforms to large scale. Information about profiling and traces of heterogeneous applications can be used to improve current JMS, which require a better knowledge of the applications~\citep{JMR:Ruiz:2014}. To predict execution or communication times in heterogeneous applications is a great challenge, for this reason, it always is meritorious to research more effective mechanisms to obtain a better performance prediction.

This thesis is related to the modeling and performance prediction of applications executed in GPUs using approaches of analytical model and machine learning techniques. Considering that the market has been taken by NVIDIA graphics cards, our models and experimentation are done on GPUs manufactured by NVIDIA. We evaluated our model using 9 developed kernels and other kernels from Rodinia Benchmark suite~\citep{Rodinia:Che:2009}, Rodinia has been used and accepted by the GPGPU community~\citep{che:2010:Rodinia, nagasaka:2010:Rodinia}, and is used in various GPU simulators~\citep{power:2014:gem5-gpu}.

These CUDA kernels were executed over GPUs with different architectures among these architectures are Kepler, Maxwell, and Pascal. We showed by using profile information for a single board, that the models are general enough to predict the execution time of an application with different input sizes and on different boards.

This thesis also presents the creation of a benchmark, which is freely available in Standard Workload Format (SWF), consisting of five applications of dense linear algebra from the Chameleon software~\citep{agullo2012morse} and a \emph{fork-join} application generated using GGen~\citep{GGen:simutools10}, this benchmark was used for ~\cite{mommessin:2017, mommesin:2017:CCPE} in the experimental evaluation of on-line and off-line scheduling algorithms in machines with heterogeneous resources.

This benchmark is available online in a web-based hosting service for version control using git, inside the repository, there are a set of application workloads with their respective execution times and dependencies, researchers can use it to test scheduler algorithms of task-based applications over CPU and GPUs.

%Our model predictions were within $XXXX$ to $XXXX$ times the measured execution times, which are reasonable for such a simple model. These results indicate that the model is good enough to generalize the predictions for different problem sizes and GPU configurations.

\section{Objectives and Thesis Contributions}
The main propose of this work is to present an analysis and design of techniques to predict the performance of applications executed on General-purpose Graphic Processing Units.

In this thesis, we first present an analysis and design of an easy and simple BSP-based model to predict the performance of GPU applications~\citep{amaris:2015:HiPC}. The model is based on the number of computations and memory accesses of the GPU, with additional information on cache usage obtained from profiling. Scalability, divergence, the effect of optimizations and differences of architectures are adjusted by a single parameter.

We also evaluate the use of different machine learning techniques to reach the same objective. We compare three different machine learning approaches: linear regression, support vector machines and random forests with our BSP-based analytical model~\citep{amaris:2016:NCA}. Our main contribution was showing that machine learning techniques provided acceptable predictions for all the applications over all the GPUs. Although the analytical model provided better predictions for some applications, it requires a deep analysis and knowledge of the applications and hardware structure. Consequently, machine learning techniques can be useful for deploying automated on-line performance prediction for scheduling applications on heterogeneous architectures with GPUs, whenever a large data set with information about similar applications is available.

Finally, we developed a benchmark of task-based applications executed on heterogeneous machines. These files are saved in format SWF (Standard Workload Format). This format was defined in order to use a pattern accepted and well used for the community in scheduling research. The benchmark is collected of six parallel applications. Five of them have been created from Chameleon, a dense linear algebra software which is part of the MORSE project~\citep{agullo2012morse}, while the sixth is a fork-join application generated automatically with GGen, a library for generating directed acyclic graphs~\citep{GGen:simutools10}. This benchmark has been tested for \citep{mommessin:2017, mommesin:2017:CCPE}.

This work was presented in the Doctoral Showcase of the International Conference for High Performance Computing, Networking, Storage and Analysis of 2017. In this conference, a summary of the thesis was presented in a short talk and poster.

We list the publications directly related to this thesis. %Each one of these papers can be found in the appendix~\ref{app:papers} in the same order as they are presented below.

\begin{itemize}

\bibitem[Amaris {\rm{\em et~al.}} (2015)Amaris, Cordeiro, Goldman, e
  Camargo]{amaris:2015:HiPC}
{\bf Marcos Amaris}, Daniel Cordeiro, Alfredo Goldman e Raphael~Y.de Camargo.
\newblock A simple bsp-based model to predict execution time in gpu
  applications.
\newblock In \emph{High Performance Computing (HiPC), 2015 IEEE 22nd
  International Conference on}, p\'{a}ginas 285--294.

\bibitem[Amaris {\rm{\em et~al.}} (2016)Amaris, de~Camargo, Dyab, Goldman, e
  Trystram]{Amaris:2016:NCA}
{\bf Marcos~Amaris}, Raphael~Y.de Camargo, Mohammed~Dyab, Alfredo~Goldman e Denis~Trystram.
\newblock A comparison of GPU execution time prediction using machine learning
  and analytical modeling.
\newblock In \emph{2016 IEEE 15th International Symposium on Network Computing
  and Applications (NCA)}, pages 326--333.

 \bibitem[Mommessin {\rm{\em et~al.}} (2017)Mommessin, Amaris, Lucarelli, e
  Trystram]{Mommessin:2017}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on hybrid multi-core
  machines.
\newblock In \emph{2017 Euro-Par: International Conference on Parallel and
  Distributed Computing}, pages 220--231.

  \bibitem[Mommessin {\rm{\em et~al.}} (2018)Mommessin, Amaris, Lucarelli, e
  Trystram]{mommesin:2017:CCPE}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on heterogeneous multi-core platforms
\newblock \emph{arXiv preprint arXiv:1711.06433} in revision. Submmited in
\emph{Concurrency and Computation: Practice and Experience}
\end{itemize}

There are other works which were published during the Doctorate of the candidate and are indirectly related to this thesis:
%\cite{bruel:2015:WSCAD}\cite{gonccalves:2016:HICSS}\cite{okada:2015:WSCAD}\cite{bruel:2017:CCPE}

\begin{itemize}
  \bibitem[Okada {\rm{\em et~al.}} (2015)Okada, Amar{\'\i}s, e vel
  Lejbman]{okada:2015:WSCAD}
Thiago~Kenji Okada, {\bf Marcos Amar{\'\i}s} e Alfredo~Goldman vel Lejbman.
\newblock Scheduling moldable bsp tasks on clouds.
\newblock In \emph{WSCAD'15 - XVI Simp{\'o}sio em Sistemas Computacionais de
  Alto Desempenho},

    \bibitem[Bruel {\rm{\em et~al.}} (2015)Bruel, Amar{\'\i}s, e
  Goldman]{bruel:2015:WSCAD}
Pedro Bruel, {\bf Marcos Amar{\'\i}s} and Alfredo Goldman.
\newblock Autotuning gpu compiler parameters using opentuner.
\newblock In \emph{WSCAD'15 - XVI Simp{\'o}sio em Sistemas Computacionais de
  Alto Desempenho}, pages\'{a}ginas 13--23.

  \bibitem[Gon{\c{c}}alves {\rm{\em et~al.}} (2016)Gon{\c{c}}alves, Amar{\'\i}s,
  Okada, Bruel, e Goldman]{gonccalves:2016:HICSS}
Rog{\'e}rio Gon{\c{c}}alves, {\bf Marcos Amar{\'\i}s}, Thiago Okada, Pedro Bruel and
  Alfredo Goldman.
\newblock Openmp is not as easy as it appears.
\newblock In \emph{49th Hawaii International Conference on System Sciences}, January-2016,
  pages 5742--5751.

\bibitem[Bruel {\rm{\em et~al.}} (2017)Bruel, Amar{\'\i}s, e
  Goldman]{bruel:2017:CCPE}
Pedro Bruel, {\bf Marcos Amar{\'\i}s} and Alfredo Goldman.
\newblock Autotuning cuda compiler parameters for heterogeneous applications
  using the opentuner framework.
\newblock \emph{Concurrency and Computation: Practice and Experience}
    29\penalty0 (22). 2017
\end{itemize}

\section{Thesis Outline}
This document is organized into 6 chapters. In Chapter~\ref{chap:characterization}, we show a detailed analysis and characterizations of GPU applications, Chapter~\ref{chap:BSPmodel} shows our proposed BSP-based model used to predict execution times of GPU applications. Chapter~\ref{Chap:ML} presents a machine learning approach for predicting execution times and compare the predictions with those of the analytical model. Chapter~\ref{Chap:Bench} describes the creation of a benchmark of task-based applications executed over heterogeneous resources, CPU and GPUs. Finally we present the conclusions and future works in the Chapter~\ref{chap:conclusion}


% !TeX spellcheck = en_GB
\section{Optimal Estimation Retrieval Algorithm} %Section - 1.2
\label{sec:retrieval}
The purpose of this study is to apply an optimal-estimation snowfall retrieval on ground based measurements to estimate the surface accumulation and vertical snow water content for an extreme event during Christmas 2016. These will later be used to compare to \SI{48}{\hour} MEPS model forecasts to see if the model was able to predict synoptical features and precipitation related to the extreme event 'Urd' in 2016.
% maybe start out by saying that you are trying to estimate atmospheric parameters  (SWC in our case) from a set of measurements.   The optimal-estimation approach provides a flexible retrieval framework that can incorporate multiple measurements/ pieces of information into a common retrieval framework.  In this way, we can use measurements from MRR, PIP, and MASC to estimate vertical profiles of snowfall.  You could then discuss the reindeer -> tracks thing to illustrate the general principle.
% literally taken from paper until Skofronick, Noh
The quantitative estimation of snowfall at the global scale from spaceborne measurements has been available only recently. Initial retrieval approaches were based on passive microwave measurements \citep{skofronick-jackson_physical_2004,noh_development_2006}. But since these passive measurements can only assess total integrated snow water path for a given column, such efforts were unable to provide much information on the profiles of snow water content.
The launch of the CloudSat \SI{94}{\giga\hertz} Cloud Profiling Radar (CPR) in 2006, however, provided the first opportunity to examine such vertical structure at a global scale.
%Several studies, such as \citet{matrosov_modeling_2007} and \citet{kulie_utilizing_2009}, have shown that the CPR can be used to estimate snofall rate but that estimated snowfall values depend heavily upon assumed snowflake microphysical properties.
Several studies have shown, that estimated snow rate depends upon retrieval assumptions such as snowflake habit and particle size distribution (PSD) and can give large differences for a given radar reflectivity.
\\
For the operational CloudSat snowfall retrieval scheme (2C-SNOW-PROFILE), \citet{wood_microphysical_2015} developed snowflake particle models based upon video snow disdrometer observations from the Canadian CloudSat-CALIPSO Validation Project \citep[C3VP,][]{hudak_canadian_2006}. Scattering properties for these snow particle models were based upon the Discrete Dipole Approximation (DDA) method.
%It was hoped that the use of realistic snow properties in the retrievals would lead to reasonable estimates of snowfall in the retrieval.
In addition, they derived an a priori relationship between particle size distribution and temperature that they could use as an additional constraint for the snowfall scheme. Use of the flexible optimal-estimation retrieval framework allowed to develop a best estimate of snow properties that are consistent with both the CPR reflectivities and the a priori constraint.
\\
They have also been used to estimate snowfall in remote locations such as the Antarctic and Arctic \citep{palerme_how_2014,kulie_shallow_2016} that in turn have been used to evaluate the representation of snowfall in climate models \citep{palerme_evaluation_2017,christensen_arctic_2016}.
These estimates have been used to assess the performance of ground-based radar schemes such as those based upon the operational weather radar system in Sweden \citep{norin_intercomparison_2015}. Despite such progress, however, the CloudSat scheme can still lead to uncertainties in the retrievals of up to \SIrange{140}{200}{\percent} \citep{wood_estimation_2011} for individual storms.
\\
Again, these uncertainties arise from the large variance in snowflake microphysical  properties as observed in nature. In response, \citet{cooper_variational_2017} explored the use of in-situ, event specific  observations of snowflake microphysical properties to improve radar-based retrievals of snowfall. This work was based upon observations from the Ka-band ARM (The Atmospheric Radiation Measurement) Zenith Radar (KAZR) and Multi-Angle Snow Camera (MASC) deployed at the ARM Climate Facility Site at Barrow, Alaska in Spring 2014. This ground-based \SI{35}{\giga\hertz} retrieval scheme was modified from the space-borne \SI{94}{\giga\hertz} CloudSat retrieval scheme developed by \citet{wood_estimation_2011}. But instead of using a temperature dependent a priori characterisation of PSD, \citeauthor{cooper_variational_2017} introduced the in-situ observations of particle size distribution through the a priori terms of the optimal-estimation framework.
\\
%
Preliminary analyses suggested good performance for this retrieval scheme at Barrow, Alaska. Estimates of snowfall from the \citet{cooper_variational_2017} approach differed by \SI{18}{\percent} relative to nearby National Weather Service snow gauge measurements for total accumulation over multiple snow events. However, given limited snowfall observed at Barrow during the deployment of the MASC, it was difficult to come to any definitive conclusions about retrieval performance. The NSF (National Science Foundation) funded field campaign with MRR, MASC, and PIP (Precipitation Imaging Package) deployment at Haukeliseter provides an ideal opportunity to further explore the \citet{cooper_variational_2017} retrieval approach. This thesis will continue to examine the sensitivity of retrieval surface snowfall rate to assumptions of habit, fall speed, and particle size distribution as in \citet{cooper_variational_2017}. In addition, this study here will will examine the vertical profiles of snowfall profiles in the atmospheric column.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\newpage
%%%% OE scheme %%%%%%%
\subsection{Snowfall Retrieval Scheme}\label{sec:ret_scheme}
%%% image forward problem %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{./images_tex/retrieval_dragon_track}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The optimal-estimation snowfall retrieval scheme was modified for the Barrow instrumentation described in \citet{cooper_variational_2017} to the \SI{24}{\giga\hertz} MRR, MASC, and PIP located at Haukeliseter. This scheme was then used to derive surface snowfall rates and vertical profiles of snow water content using different combinations of retrieval assumptions based upon in-situ observations.
Here a discussion of the optimal-estimation framework is presented so that the reader can understand exactly how the different measurements in the retrieval scheme were incorporated. The differences in the forward and inverse problem are reviewed in the beginning. Both, forward and inverse problem underlie the basic methodologies of remote sensing.
\par\medskip\noindent
The general concepts of forward and inverse problems are illustrated in \Cref{fig:dragon}.  The forward problem describes the relationship between the physical parameters of interest and the measurements (\Cref{fig:dragon:forward}). In \Cref{fig:dragon:inverse}, the physical parameter is the reindeer and the measurements are its tracks. In the here presented thesis snowfall scheme, the physical parameters are the vertical profiles of snow water content (SWC) and the measurements are the MRR reflectivities.  The inverse problem represents the opposite goal.  The physical entity (reindeer and SWC) must be inferred from the measurements (reindeer tracks or MRR reflectivities).
\\
The optimal-estimation framework is used to solve the inverse problem for the thesis work. But it is more complex than the direct inversion as represented in \Cref{fig:dragon:inverse}. Instead of inverting directly for the parameter of interest (SWC) from the measurements [\SI{}{\decibel ze}], the scheme includes additional information based upon an 'a priori' understanding of the physics of the retrieval problem. Thus, it represents a weighted balance between what the data can tell about the state and what is already known about it. For the CloudSat retrieval scheme, this a priori information came from a parametrisation relating particle size distribution (PSD) to temperature. For the Barrow and Haukeliseter schemes, the a priori information could also come from the in-situ observations of snowflake microphysics. These constraints also add numerical stability to the inversion process when there are either calibration errors in the measurements or uncertainties in the forward model that relates the physical parameter to measurement space. Details of the radar forward model are discussed at the end of this section.
\\
The optimal estimation method is based on Gaussian statistics.  It solves for snowfall properties of interest or retrieval vector, $\mathbf{x}$, by minimizing the scalar cost function, $\Phi$ , as in \Cref{eq:scalar_cost_fct}.
\begin{equation}
\begin{split}
\Phi(\mathbf{x},y,a) = & (y- F(\mathbf{x}))^T \mathbf{S}_y^{-1} 			(y-F(\mathbf{x})) \\
&+(\mathbf{x}-a)^T \mathbf{S}_{a}^{-1} (\mathbf{x}-a)
\end{split} \label{eq:scalar_cost_fct}
\end{equation}
Specifically, or this thesis retrieval scheme, $\mathbf{x}$ represents the PSD parameters of slope and number intercept of an assumed exponential size distribution for each radar range bin as in \Cref{eq:num_dens}. $y$ is the vector of MRR reflectivites. The vector $a$ is the a priori guess for slope parameter and number in each range bin. $F(\mathbf{x})$ represents the forward model that translates snow properties into reflectivity space.  Minimizing the cost function therefore seeks to reduce the difference between the observations, $y$, and simulated observations, $F(x)$ and between the a priori guess ($a$) and the retrieval vector ($x$).
\begin{align}
	n(r) & = N_{0} \exp\left(-\lambda r\right) \qquad [ \SI{}{\per\cubic\metre\per\mm} ] \label{eq:num_dens}
\end{align}
The $\mathbf{S}_y$ and $\mathbf{S}_{a}$ terms, in \Cref{eq:scalar_cost_fct}, represent the forward model and measurement error covariance matrix and the a priori error covariance matrix, respectively. The relative differences between $\mathbf{S}_{a}$ and $\mathbf{S}_y$ weight the importance of the observations and the a priori considerations in determining our best estimate of PSD properties.
\\
Newtonian iteration is used until the value of the cost function converges and our best estimate of snowfall properties are found.  The optimal-estimation scheme also provides error diagnostics through the retrieval error covariance matrix, $\mathbf{S}_x$, as in \Cref{eq:Sx}.
\begin{align}
	\mathbf{S}_x & = \left( \mathbf{S}_a^{-1} + \mathbf{K}^T \mathbf{S}_y^{-1} \mathbf{K} \right)^{-1}\label{eq:Sx}
\end{align}
The Jacobian matrix, $\mathbf{K}$, represents the sensitivity matrix of the perturbed result of the forward model. The true state $\mathbf{x}$ is perturbed by \SI{0.2}{\percent} and thus $\mathbf{K}$ represents the relation between simulated values to the true state and how sensitive the simulated values are to small changes when starting a new retrieval cycle. The closer $\mathbf{K}$ is diagonal, the more is $\mathbf{x}$ determined by the real observed and a priori values. If the limit of the partial derivative is close to unity, the retrieved value $\mathbf{x}$ is its true state \citep{wood_estimation_2011}.
\\
In practical application for this multiple layer retrieval scheme, log-transformed particle size distribution parameters of slope and number intercept were used due to the large expected range of these variables. The state vector, $\mathbf{x}$, is defined in \Cref{eq:snow_prop}.
\begin{align}
	\mathbf{x} & = \begin{bmatrix}
		log(\lambda)_0 	\\
		\vdots 			\\
		log(\lambda)_{\text{nlayer}} 	\\
		log(N_0)_0		\\
		\vdots			\\
		log(N_0)_{\text{nlayer}}
	\end{bmatrix} \qquad \text{nlayer} = 14
	\label{eq:snow_prop}
\end{align}
The usage of a priori terms were explored both from in-situ microphysical observations and from the PSD-temperature relationship developed for the CloudSat scheme as in \Cref{eq:lambda,eq:N0}. Temperatures in \SI{}{\celsius} at Haukeliseter were taken from site measurements with an assumption of a moist adiabatic lapse rate for the observed snow events. The log transformed slope and number intercept values were taken from \citet{wood_estimation_2011}.
\begin{align}
	\log(\lambda) & = -0.03053 \cdot T_{ap} - 0.08258  \label{eq:lambda} \qquad [ \log(\SI{}{\per\mm}) ] \\
	\log(N_0) & = -0.07193 \cdot T_{ap} +2.665  \qquad [ \log(\SI{}{\per\cubic\metre\per\mm})]
	\label{eq:N0}
\end{align}
The log-transformed equations are useful, since the results from C3VP were similar to other observations. The study showed, that $N_0$ ranges over several order of magnitude as well as $\lambda$ was non-Gaussian for the snow events \citet{wood_estimation_2011}. The diagonal matrix elements in $\mathbf{S}_a$ (\Cref{eq:scalar_cost_fct,eq:Sx}) are equal to \numlist{0.133;0.95} for the particle slope parameter and the number intercept, respectively, as from Eq. 7.35 and 7.36 in \citet{wood_estimation_2011}. The diagonal matrix elements for $\mathbf{S}_y$ are $2.5^2$ in \Cref{eq:scalar_cost_fct,eq:Sx}.
\\
After the best estimate of PSD parameters are found, the snow water content in each layer is calculated using the snow particle mass-dimension relationships as in \Cref{app:scat_scheme}.
%
\begin{align}
	\text{SWC} & = \int_{r_{min}}^{r_{max}} m(r) n(r) dr \qquad [\SI{}{\gram\per\cubic\metre}] \label{eq:SWC}
\end{align}
$r$ is the particle maximum dimension and $m(r)$ the related mass.
\par\medskip\noindent
This thesis work considered the database of particle models developed for the CloudSat mission, e.g. different types of aggregates, sector plates, and columns. Scattering properties for these snowflakes were calculated for the \SI{24}{\giga\hertz} frequency using discrete dipole approximation (DDA). Observations from the MASC of snowflake habit were used to guide particle selection. Snow water content, in turn, was translated into a snowfall rate using fall speed observations (MASC, PIP, or MRR Doppler velocity) or climatological analyses ($V = \SI{0.85}{\mPs}$). Surface snowfall rate is estimated using the SWC from the lowest non-noise reflectivity and radar bin.
\\
The forward model that calculates simulated \SI{24}{\giga\hertz} MRR reflectivities from PSD parameters was modified from that used in the CloudSat \SI{94}{\giga\hertz} operational snowfall product (2C-SNOW-PROFILE). Backscatter from frozen hydrometeors in each radar bin is summed up as in \Cref{eq:backscatter} form which reflectivity factor, Z, can be found (\Cref{eq:singleZ}).
% singly-scattered non-attenuated reflectivity
\begin{align}
	\eta_{bk} & = \int_{r_{min}}^{r_{max}} n(r) \sigma_{bk} dr \qquad [\SI{}{\per\metre}] \label{eq:backscatter} \\
	Ze^{ss,na} & = \frac{\Lambda^4}{\left\| K_w \right\|^2 \pi^5} \eta_{bk} \qquad [\SI{}{\mm^6\metre^{-3}}] \label{eq:singleZ}
\end{align}
Where, $\Lambda$ is the wavelength of the radar; $\left\| K_w \right\|^2$ is the complex refractive index of water. Radar backscatter values were estimated using discrete dipole approximation for the CloudSat particle models at \SI{24}{\giga\hertz}. Unlike the \SI{94}{\giga\hertz} spaceborne CloudSat mission, multiple scattering and attenuation can be neglected for \SI{24}{\giga\hertz} MRR and the short path length retrievals as viewed from the ground perspective.

%\newpage
\subsection{Environmental Masks for the Optimal Estimation Retrieval}\label{sec:pre_snow}
Different steps and assumptions are done in the here presented snowfall retrieval, to achieve vertical profiles of snowfall from MRR. The snowfall rate at the surface can be estimated from one of the lower levels. The optimal estimation retrieval is only performed for profiles, which are likely to have observed snow.
%Relationships between reflectivity and snowfall have been developed in previous studies. Different crystal shapes led to different results, even if the PSD of ice particles is known. Snow densities vary significantly from storm to storm, where small particles are still Rayleigh scattered, and larger particles non-Rayleigh scattered \citep{gunn_microwave_1954}.
%%%% image surface temperature and MRR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{./images_tex/retrieval_MRR_sfcT}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent
\\
This value was  chosen as sensitivity studies \citep[e.g.][]{wood_level_2013} that a days worth of such reflectivities would produce only a trace of snow.  Such a value therefore guarantees that a significant snow event is not missed and that any storms with lower \SI{}{\decibel Z} values would not produce meaningful precipitation.
%A reflectivity threshold of \SI{- 15}{\decibel Z} was selected for the scheme. The threshold is similar to the one used in \citet{wood_level_2013}. Light liquid precipitation is related to \SI{- 10}{\decibel Z} \citep{stephens_properties_2007}. \citet{wood_estimation_2011} found, that the reflectivity above \SI{-15}{\dB Z} are not influenced by ground clutter, when the reflectivity in the lowest bin and adjacent bin is compared.
The Haukeliseter measurement site is equipped with a weather mast, measuring the air temperature every minute at two-meter height (compare \Cref{fig:MRR_sfcT}, upper panel).
Since the MRR measures above \SI{300}{\metre} and temperature measurements exists only at the surface, a priori temperature ($T_{ap}$) at the surface is assumed to be similar to the observed near-surface air temperature. The use of a moist adiabatic lapse rate of $dT/dz = \SI{5}{\kelvin\per\km}$ gives $T_{ap}$ in each layer.
Snow existence at temperature measurements up to a threshold of +\SI{2}{\celsius} are assumed. \citet{liu_g._deriving_2008} validated this threshold, by analysing present weather reports to find the distinction between liquid and solid precipitation.\\
The purple line in the lower panel of \Cref{fig:MRR_sfcT} represents the time frame during \SI{25}{\dec}, where the MRR reflectivity is less than \SI{- 15}{\decibel Z}, and a priori temperature passes the \SI{2}{\celsius} limit at the surface.


\subsection{Datasets}
\begin{frame}
  \frametitle{Modeling the University of Illinois}
  \begin{itemize}
    \item All data is from the University of Illinois
    \item UIUC is a good model for thinking about hybrid energy systems.
    \begin{itemize}
      \item Solar Power
      \item Wind Power
      \item Natural Gas
      \item District Heating
    \end{itemize}
  \end{itemize}
\end{frame}

\subsection{Echo State Networks}
\begin{frame}
  \frametitle{Echo State Networks}
  \begin{figure}
    \includegraphics[width=\textwidth]{reservoir_network.jpg}
    \caption{A conceptual diagram of an echo state network. The reservoir is a large sparse matrix with randomly assigned entries \cite{pathak_model-free_2018, lukosevicius_practical_2012}.}
    \label{fig:resnet}
  \end{figure}
\end{frame}

\begin{frame}
  \frametitle{Hyper-parameter Optimization}
  \begin{columns}
    \column[t]{5cm}
      \begin{figure}
        \includegraphics[width=\columnwidth]{solar-angle-gridsearchNP.png}
        \caption{A grid search for the set of noise and spectral radius, $\rho$, that minimizes the mean squared error.}
        \label{fig:gridsearch}
      \end{figure}
    \column[t]{5cm}
      \vspace{1cm}
      \begin{align}
        MSE = \frac{1}{N}\sum_i^N(\hat y - y_i)^2
      \end{align}

      \begin{itemize}
        \item The optimal set of parameters is ``reservoir specific''
        \item Several other parameters need to be optimized such as:
          \begin{itemize}
            \item Reservoir Size
            \item Sparsity
            \item Training Length
          \end{itemize}
      \end{itemize}
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Uncertainty Analysis}
  \begin{figure}
    \centering
    \includegraphics[width=0.85\textwidth]{demsol-bars01.png}
    \caption{Total demand prediction with error bars. The 60\% confidence is $\pm 1\sigma$ and the 95\% confidence interval is $\pm 2\sigma$. Mean is generated from predictions made by several different reservoirs.}
    \label{fig:uncertainty}
  \end{figure}
\end{frame}


\section{Materials and methods}

\subsection{Data}

We consider an online community called Edgeryders, whose members discuss and try to implement various projects connected to social innovation in a decentralized manner. The community is run by a British company, also called Edgeryders, which we cast as the policy maker of the community. The affordances of the online platform that hosts the community and its discussion deeply influence the social dynamics thereof (\cite{hodas2014simple}). Those of Edgeryders are as follows: the discussion is hosted on a web platform based on an open source framework called Drupal 7. Drupal is one of the largest open source projects globally; in this sense, Edgeryders is fairly typical from a technology point of view. It consists of users posting reflections or status reports on their social innovation projects; these are then commented upon by ot her members of the community. The Edgeryders community website supports what is known in online community parlance as threaded comments: in other words, comments are themselves commentable. Of course, posts are commentable too, so that a comment can be directed either towards a post or towards another comment. Both posting and commenting are restricted to registered users only. This architecture has not changed since inception of Edgeryders in October 2011.

\begin{figure}
	\includegraphics[width= 1 \textwidth]{Threaded_comments_Edgeryders}
	\caption{Threaded comments in the Edgeryders online community. Each comment is in turn commentable. This is represented visually by nested indentation: "sibling" comments (comments to the same post or comment) are visualized in chronological order, with the same level of indent. "Children" comments are visualized directly under their "parent", indented with respect to it.}
	\label{fig:threadedComments}
\end{figure}

We extracted a snapshot of the database on October 7th 2015. At the time, the community had 2,904 registered users, who had authored 4,062 posts and 18,285 comments. 23 of the users had, at some point in time, reported to the Edgeryders company; we cast them as the online community managers for this particular community.

All events in our dataset (creating accounts, creating posts, creating comments) are timestamped. This allows for the possibility of modelling interaction in the online community dynamically.

\subsection{A network model of interaction in online communities}

We need to consider carefully how we incorporate dynamics in our model (\cite{holme2012temporal}). We make the following assumptions:

\begin{enumerate}
	\item The duration of interactions is negligible. This means that events can be represented by a contact sequence, i.e. triples $(i, j, t)$ where $i, j \in V$, the set of interactants (represented as vertices in the network) and $t$ denotes time. This is a standard assumption in studies of person-to-person communication online (\cite{holme2012temporal}).
	\item First-time interaction produces a permanent change in the social relationship between the source and the target of the interaction (the author and the recipient of the comment). All Edgeryders users can be assumed to share some common interests around social innovation, that motivated them to join Edgeryders. When they interact directly, however, they begin to unpack and specify such common interests, and they (often explicitly) signal interest in each other's activities. This produces a shift in the nature of the relationship between the interacting, which becomes one of actual, as opposed to potential, interaction. Mathematically, this translates into representing interaction by a network whose edges do not decay.
	\item Subsequent interaction also carry social significance. They have the effect of further strengthening the relationship between the two interactants. The strength of the relationship increases monotonically with the number of interactions recorded. The effect of all  interactions after the first one, too, is permanent. Mathematically, this translates into representing interaction by a weighted network.
\end{enumerate}

We then proceeded to build a social network of the interactions within the community as follows:

\begin{enumerate}
	\item Users that have posted at least one post or one comment are included as nodes in the network.
	\item We interpret each comment as one interaction event. Whenever a comment is posted, an edge is induced. The new edge's source node is the node associated to the author of the comment. The new edge's target node is the node associated to the author of the post or comment being commented.
	\item If an edge with the same source and target already exist, we do not add a new one, but rather increase the edge's weight by one.
	\item If the source and target of the edge are the same, the edge is discarded.
	\item The sequence of timestamped interaction events allows us to model interaction in Edgeryders as an evolving network. We divide the observation period in one week-long intervals, and consider users to be acting on the basis of the interaction network that they find themselves in at that period. In other words, we do not attempt to model explicitly the entire sequence of interactions; rather, we assume that all agents make simultaneous decisions on who to interact with at the beginning of each time period, and that those decisions are based on the snapshot of the network at the beginning of said period. This means we are following the tradition of literature on evolving networks (\cite{dorogovtsev2002evolution}) rather than that of temporal networks (\cite{holme2012temporal}). This reflects our preoccupation with topological aspects over temporal ones. We claim the choice is justified under the terms suggested by Holme and Saramaki (\cite{holme2012temporal}) \footnote{" [...] a contact sequence that is fairly well modeled by a weighted graph with the assumption that contact times are random, with a frequency proportional to the edge weight"} .

\end{enumerate}

This process results in a sequence of 212 networks, each one a step on the growth path of the interaction network in Edgeryders. At the end of week 1 the network had only 6 nodes and 1 edge; at the end of week 212 it had 789 nodes and 4,861 edges.

\begin{figure}
	\includegraphics[width=.8 \textwidth]{numnodes}
	\includegraphics[width=.8 \textwidth]{numedges}
	\caption{Growth of the number of nodes (top) and edges (bottom) in the Edgeryders interaction network.}
\label{fig:growthNodesEdges}
\end{figure}

\subsection{Model}

We model user activity in an online community as a function of three groups of variables:

\begin{equation}
	A_{i,t} = f(NN_{i,t}, EN_{i,t}, GN_t)
	\label{eq:model}
\end{equation}

Where:
\begin{itemize}
\item $A_{i,t}$ denotes the activity of user $i$ at time $t$, defined as the number of posts plus the number of comments authored by user $i$ during the period.
\item $NN_{i,t}$ denotes a vector of non-network variables associated with user $i$ at time $t$. These include writing a post or a comment; receiving a comment from a community manager; receiving a comment from another user who is not a community manager; the total number of posts and comments written by other users users $A_{j,t}, j \neq i$; and $i$'s experience as a user of Edgeryders, as measured by the number of weeks elapsed since she joined the community. The week of joining can predate the week in which the user writes her first post or comment.
\item $EN_{i, t}$ denotes a vector of variables pertaining to the user's ego network at time $t$. These include her own in-degree (the number of users linking to her); her out-degree (the number of users she links to); her betweenness centrality (the fraction of shortest paths across any two users $j,k \neq i$ that $i$ finds herself on); her PageRank (the probability that a random walker across the network would end up at this particular node\footnote{PageRank was originally developed as a measure of network centrality for the World Wide Web (\cite{brin2012reprint}).}; and her clustering coefficient (the fraction of $i$'s neighbours that are also neighbours to each other).
\item $GN_{i,t}$ denotes a vector of variables pertaining to the global network at time $t$. These include the total number of nodes; the total number of edges; the average degree; and the network's Louvain modularity\footnote{The modularity of a network is a measure of how much it differs from an Erdos-Renyi random network with the same degree distribution. Values close to 0 indicate it is indistinguishable from a random network; values close to -1 or 1 indicate structure (\cite{clauset2004finding}). Measuring modularity is computationally hard; it is customary to use algorithms to compute an approximate value. Of these, the Louvain algorithm is the most widely used in the literature for its attractive computational efficiency \cite{blondel2008fast}.  }.
\end{itemize}

\subsection{Estimation}

We estimate the behaviour of a binary variable $A_{i,t}$ that takes value $1$ if user $i$ engages actively in interaction (i.e. writes a post or a comment at time $t$), and $0$ otherwise. We make use of a logit model:

\begin{equation}
	PR(A_{i,t}= 1|X_{i,t}) = \frac{e^{X_{i,t}\beta}}{1 + e^{X_{i,t}\beta}}
	\label{eq:estimation}
\end{equation}

Where $X_{i,t}$ is a vector of observed explanatory variables; $\beta$ is a vector of parameters.

An advantage of the logit model is that its coefficient lend themselves to being interpreted in terms of marginal effects on the log-odds ratio (\cite{cameron2005microeconometrics}). We rewrite equation \ref{eq:estimation} as

\begin{equation}
	ln \frac{Pr(A_{i,t} = 1)}{1 - Pr(A_{i,t}=1)} = X_{i,t} \beta
	\label{eq:logOdds}
\end{equation}

Next, we proceed to estimating equation \ref{eq:logOdds} with fixed effects on the 766 Edgeryders users who have no moderator or site administrator role. To do this, we drop the 23 users that have such roles. We add to the right hand-side
$c_i$, an unobserved time-invariant effect specific to user $i$; this is intended to get rid of the bias introduced by different individuals having a different propensity to participate in the online conversation, as this is likely to be correlated with some of the regressors\footnote{For example, it is possible that "chattier" individuals might end up with higher in- and -out degrees, higher network centrality, and so on. }. We also add $u_{i,t}$, a residual error term, with mean zero and uncorrelated with right-hand side variables. We take care to employ lagged variables when failure to do so might cause endogeneity issues.

\begin{equation}
	ln \frac{Pr(A_{i,t} = 1)}{1 - Pr(A_{i,t}=1)} = X_{i,t} \beta + c_i + u_{i,t}
	\label{eq:logOddsErrors}
\end{equation}

\subsection{Software stack}
Primary data from the Edgeryders database were obtained using a Drupal plugin called Views JSON. We used a modified version of an application software called Edgesense\footnote{\url{https://github.com/Wikitalia/edgesense}. Modifications added support for computing some extra network metrics, like the clustering coefficient.} to build a network representation of the interaction in Edgeryders. We then enriched the data so obtained with non-network metrics computed directly from the primary data, and exported the results in tabular form. Such results were then imported into Stata for estimation.

Code and data are available at the following address: (redacted to guarantee author's anonymity).